2. CD Concepts and Strategies

Overview

In the previous capsule you saw the difference between CI and CD at a general level. Now you go deeper into the fundamental concepts of Continuous Delivery: what it is, how it differs from Continuous Deployment, when to use each approach, and how a deployment pipeline is structured. This distinction isn't academic — it defines how you design your pipeline and what level of automation you apply.

For AI systems, the choice between Delivery and Deployment has direct implications. A model that generates incorrect responses in production doesn't just cause bugs — it can generate significant costs in API calls, produce outputs that confuse users, or violate business constraints. That's why most teams working with AI choose Continuous Delivery (with human approval) over Continuous Deployment (fully automatic). The human in the loop isn't weakness — it's prudence.

This capsule gives you the conceptual framework for understanding the deployment pipeline you build in the rest of the module.

Continuous Delivery vs Continuous Deployment

Continuous Delivery

The definition: Every change that passes the tests is ready to go to production, but it requires a manual action (a click, an approval) to be deployed.

Push → CI Pipeline → The image is ready → ⏸️ Human approval → Deploy to production
                                               ↑
                                          "Is it OK?"

The pipeline goes as far as "the image is in the registry and staging works." But it doesn't deploy to production automatically. A human reviews, verifies staging, and approves.

The characteristics:

• 📋 Always deployable — Every commit that passes CI can go to production
• 📋 Human control — Someone decides when it gets deployed
• 📋 Variable frequency — You can deploy once a day or once a week
• 📋 A safety net — If staging shows something unexpected, you don't deploy

Continuous Deployment

The definition: Every change that passes the tests gets deployed to production automatically. There's no manual step.

Push → CI Pipeline → The image is ready → Deploy to production (automatic)

There's no pause. If the tests pass, the code goes to production. The pipeline is completely autonomous.

The characteristics:

• 📋 Fully automated — From push to production with no intervention
• 📋 Maximum speed — Changes reach users in minutes
• 📋 It requires total confidence in the tests — If the tests don't catch a bug, production suffers it
• 📋 Fast feedback — You discover problems in production immediately

The direct comparison

Visually

Continuous Integration (CI):
  Code → Build → Test → ✅ "The code is valid"
  
Continuous Delivery (CD - Delivery):
  Code → Build → Test → Deploy Staging → ⏸️ Approval → Deploy Prod
                                              ↑
                                       A human reviews
  
Continuous Deployment (CD - Deployment):
  Code → Build → Test → Deploy Staging → Deploy Prod (automatic)
                                              ↑
                                        No intervention

Why Continuous Delivery is better for AI systems

The main argument

AI systems have a problem that traditional applications don't: non-deterministic behavior. A CRUD always returns the same data for the same query. An endpoint that calls GPT-4 can return different responses each time, and not every failure is catchable by automated tests.

The scenario: You change your chatbot's system prompt

The automated tests:
  ✅ The endpoint returns 200
  ✅ The response is a non-empty string
  ✅ The JSON format is correct
  ✅ The tokens used are within the limit
  
But the tests do NOT verify:
  ❓ Is the response useful to the user?
  ❓ Is the tone appropriate for the context?
  ❓ Is the model not hallucinating data?
  ❓ Is the latency acceptable under load?

A human reviewing staging can detect these problems. Automated tests can't (at least not completely, not yet).

When each approach is appropriate

Use Continuous Delivery (with approval) when:

• 📋 Your application calls LLMs (GPT, Claude, etc.) — non-deterministic outputs
• 📋 You change prompts or model configuration — an impact that's hard to test automatically
• 📋 Your team is small (1-5 people) — the overhead of approval is minimal
• 📋 The costs of an error are high — unnecessary API calls, incorrect data
• 📋 You're at an early stage — your test suite doesn't cover every edge case

Use Continuous Deployment (automatic) when:

• 📋 Your test suite is very mature — >90% coverage, integration tests, load tests
• 📋 You have robust monitoring — immediate alerts when something fails in production
• 📋 The changes are low-risk — typos, UI tweaks, config changes
• 📋 Your team deploys many times a day — the overhead of approval is a bottleneck
• 📋 You have a reliable automatic rollback — you can revert in seconds

The recommendation for this guide

For your deployment pipeline:

  ✅ Continuous Delivery with an approval gate

  Push → Test → Build → Deploy Staging → ⏸️ A reviewer approves → Deploy to Production

  Why?
  1. Your test suite is under construction (it's not exhaustive yet)
  2. Your prompts change frequently (a non-deterministic impact)
  3. Production API keys cost real money
  4. The overhead of a reviewer approving is < 5 minutes
  5. The peace of mind of knowing that someone verified is worth more than the speed

When your test suite is mature and you have robust monitoring (after guide #18), you can evolve to Continuous Deployment. But starting with Delivery is the prudent decision.

The Deploy Pipeline Pattern

The anatomy of a deployment pipeline

Every deployment pipeline follows a pattern of progressive stages. Each stage adds confidence:

Stage 1: Build
  → The code compiles/builds with no errors
  → Confidence: "The code is syntactically valid"

Stage 2: Test
  → The unit and integration tests pass
  → Confidence: "The code does what we expect"

Stage 3: Package
  → The Docker image gets built and pushed
  → Confidence: "The code runs in a container"

Stage 4: Deploy Staging
  → The image gets deployed to a real environment
  → Confidence: "The code works on a real server"

Stage 5: Validate
  → Smoke tests verify basic functionality
  → Confidence: "The application responds correctly"

Stage 6: Approve
  → A human verifies and approves
  → Confidence: "An expert reviewed it and agrees"

Stage 7: Deploy Production
  → The image gets deployed to production
  → Confidence: "Users have the new version"

Stage 8: Verify
  → Smoke tests in production + monitoring
  → Confidence: "Production is stable"

The principle of progressive confidence

Each stage is more expensive to run and closer to the user. If a change is going to fail, you want it to fail as early as possible — at the cheapest stage:

                            Cost of the failure
Stage 1: Build              │█                      Low (only CI minutes)
Stage 2: Test               │██                     Low (only CI minutes)  
Stage 3: Package            │███                    Medium (build + push)
Stage 4: Deploy Staging     │█████                  Medium (the staging server)
Stage 5: Validate           │██████                 Medium (smoke tests)
Stage 6: Approve            │███████                High (human time)
Stage 7: Deploy Production  │█████████████          Very high (users affected)
Stage 8: Verify             │████████████████       Maximum (production unstable)

If your tests are good, most failures get detected in stages 1-3. Staging detects environment problems. Approval detects business problems. Production should only receive validated changes.

The Deploy Pipeline in GitHub Actions

The pipeline's basic structure

name: Deploy Pipeline

on:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements-dev.txt
      - run: pytest tests/ -v

  build:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}:sha-${{ github.sha }}

  deploy-staging:
    needs: build
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - name: Deploy to staging
        run: echo "Deploying to staging..."

  deploy-production:
    needs: deploy-staging
    runs-on: ubuntu-latest
    environment:
      name: production
    steps:
      - name: Deploy to production
        run: echo "Deploying to production..."

The needs chain

test → build → deploy-staging → deploy-production

Each job depends on the previous one. If test fails, nothing else runs. If build fails, there's no deploy. If deploy-staging fails, production doesn't get touched.

The key is environment: — when a job declares an environment with protection rules, GitHub pauses the workflow until those rules are met (the reviewer's approval, a wait timer, etc.).

The difference between the CI pipeline and the Deploy pipeline

The CI Pipeline (Modules 1-5):
  Trigger: push + PR
  Jobs: test → build → scan → push
  Output: An image in GHCR
  
The Deploy Pipeline (this module):
  Trigger: only a push to main (not PRs)
  Jobs: test → build → deploy-staging → approve → deploy-production
  Output: An application running in production

You can have them as separate workflows or as a single combined workflow. In this guide we keep them separate for clarity, but in Module 8 (the capstone project) we combine them.

Environments as pipeline stages

Mapping environments to stages

GitHub Environments map directly to your deployment pipeline's stages:

GitHub Environment: staging
  → No protection rules (an automatic deploy)
  → Secrets: OPENAI_API_KEY (a limited budget), SERVER_SSH_KEY (staging)
  → URL: https://staging.your-app.com

GitHub Environment: production
  → Protection rules: a required reviewer, a wait timer
  → Secrets: OPENAI_API_KEY (production), SERVER_SSH_KEY (production)
  → URL: https://your-app.com

Each environment has its own secrets. Staging uses an API key with a limited budget ($10/month). Production uses the real API key. If staging has a bug that causes retry loops, the costs are bounded.

The flow of secrets per environment

deploy-staging:
  environment: staging
  steps:
    - run: echo "Using staging secrets"
      env:
        OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        # Value: sk-staging-abc123 (budget: $10/month)

deploy-production:
  environment: production
  steps:
    - run: echo "Using production secrets"
      env:
        OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        # Value: sk-prod-xyz789 (no limit)

The secret's name is the same (OPENAI_API_KEY), but the value differs per environment. The workflow doesn't need to know the difference — GitHub injects the right value depending on the declared environment.

Pipeline triggers: when to deploy

Only main, never PRs

The deploy pipeline triggers only on a push to main, not on PRs:

on:
  push:
    branches: [main]

Why not on PRs: A PR is proposed code, not accepted code. Deploying code that hasn't been merged into main is dangerous — it could overwrite staging with experimental code.

Push to main + tags for releases

on:
  push:
    branches: [main]
    tags: ["v*"]

This triggers the pipeline both on a push to main (an automatic deploy to staging) and on version tags (a release deploy).

Workflow dispatch for manual deploys

on:
  push:
    branches: [main]
  workflow_dispatch:
    inputs:
      environment:
        description: "Target environment"
        required: true
        type: choice
        options:
          - staging
          - production
      image_tag:
        description: "Image tag to deploy"
        required: true
        type: string

workflow_dispatch lets you deploy manually from the Actions UI. Useful for rollbacks (deploying a previous tag) or re-deploys.

Comparisons

A complete deployment pipeline vs a simple deploy

One workflow vs separate workflows

A self-hosted runner vs a GitHub-hosted runner

Troubleshooting

1. "Environment 'staging' not found"

Symptom:

Error: Environment 'staging' was not found.

Cause: The environment isn't created on GitHub, or the name has a typo.

Solution:

GitHub → Settings → Environments → New environment
  Name: staging (exactly as in the workflow YAML)
  → Configure environment

Verify that the name in the YAML matches the environment's name on GitHub exactly (case-sensitive).

2. The deploy job stays in "Waiting"

Symptom: The job shows "Waiting for review" but there's no reviewer configured.

Cause: The environment has protection rules with required reviewers, but nobody can approve.

Solution:

GitHub → Settings → Environments → production → Protection rules
  Required reviewers: Add your username or a team

If you're practicing solo, add yourself as a reviewer — you can approve your own deploys.

3. The workflow doesn't trigger on a push to main

Symptom: You push to main, but the deploy workflow doesn't appear in Actions.

Cause: The YAML file isn't in .github/workflows/, it has a syntax error, or the branch filter doesn't match.

Solution:

# Verify that the file exists
ls .github/workflows/deploy.yml

# Validate the YAML
python -c "import yaml; yaml.safe_load(open('.github/workflows/deploy.yml'))"

# Verify the trigger
# The YAML must have:
# on:
#   push:
#     branches: [main]

4. The environment's secrets don't get injected

Symptom: The step uses ${{ secrets.MY_SECRET }} but the value is empty.

Cause: The job doesn't declare environment:, or the secret is in a different environment.

Solution:

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: staging    # NECESSARY to access staging's secrets
    steps:
      - run: echo "Key length: ${#KEY}"
        env:
          KEY: ${{ secrets.OPENAI_API_KEY }}

Without the environment: staging line, the job can't access the environment's secrets.

Exercises

Exercise 1: Identify CI vs CD

For each scenario, say whether it's CI, Continuous Delivery, or Continuous Deployment:

1. A push triggers tests and lint automatically
2. If the tests pass, the image gets pushed to GHCR
3. The image gets deployed to staging automatically
4. A reviewer reviews staging and approves
5. The image gets deployed to production after the approval
6. Every push to main deploys to production with no human intervention

Exercise 2: Design the chain of jobs

You have these jobs: lint, test, build-docker, deploy-staging, smoke-test, deploy-production. Define the needs: chain so that: (1) lint and test run in parallel, (2) docker depends on both, (3) staging depends on docker, (4) smoke-test depends on staging, (5) production depends on smoke-test.

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ruff check src/

  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pytest tests/ -v

  build-docker:
    needs: [lint, test]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}:sha-${{ github.sha }}

  deploy-staging:
    needs: build-docker
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - run: echo "Deploy to staging"

  smoke-test:
    needs: deploy-staging
    runs-on: ubuntu-latest
    steps:
      - run: curl -sf https://staging.app.com/health

  deploy-production:
    needs: smoke-test
    runs-on: ubuntu-latest
    environment:
      name: production
    steps:
      - run: echo "Deploy to production"

lint ──┐
       ├──→ build-docker → deploy-staging → smoke-test → deploy-production
test ──┘

Exercise 3: Configure the triggers for a deploy pipeline

Create the on: section of a workflow that: (1) triggers on a push to main, (2) triggers on tags starting with v, (3) can be triggered manually with an input to choose the environment (staging or production) and another for the image tag.

name: Deploy Pipeline

on:
  push:
    branches: [main]
    tags: ["v*"]

  workflow_dispatch:
    inputs:
      environment:
        description: "Target environment"
        required: true
        type: choice
        options:
          - staging
          - production
      image_tag:
        description: "Image tag to deploy (e.g., sha-abc1234 or v1.2.3)"
        required: true
        type: string

Exercise 4: Choose the strategy for each scenario

For each scenario, choose Continuous Delivery or Continuous Deployment and justify it:

1. A 3-person startup with an AI chatbot that changes prompts daily
2. A company with a mature test suite, monitoring with PagerDuty, and 50 deploys/day
3. A personal portfolio project that only you use
4. An ML team that deploys models classifying bank transactions

Summary

• ✅ Continuous Delivery = ready to deploy + human approval before production
• ✅ Continuous Deployment = an automatic deploy to production with no intervention
• ✅ For AI, Delivery is the recommendation — non-deterministic outputs need a human eye
• ✅ The deploy pipeline pattern: build → test → staging → approve → production
• ✅ Progressive confidence: each stage adds confidence, each failure is cheaper early on
• ✅ Environments map to stages: staging (automatic), production (with approval)
• ✅ Triggers: push to main for automatic deploys, workflow_dispatch for manual ones
• ✅ CI produces artifacts; CD takes them to production — they're complementary, not substitutes

Additional resources

1. Continuous Delivery vs Continuous Deployment — The difference explained by Atlassian
2. GitHub Actions — Using environments — Environments as stages
3. Martin Fowler — Continuous Delivery — The original concept by the book's co-author
4. The Twelve-Factor App — Build, Release, Run — Separating stages in modern apps
5. GitHub Actions — Workflow syntax — Reference for needs:, environment:, triggers
6. Accelerate (book) — Research on deployment frequency and stability