1. Introduction: Production AI Pipeline

Overview

Modules 1-7 gave you the individual pieces: basic workflows, automated testing, AI checks, secrets management, Docker builds, deployment pipelines, notifications, and advanced patterns. Each module produced a functional mini-project. But in production, these pieces don't exist in isolation — they need to work together, in order, with error handling between stages, and with the ability to recover from failures without manual intervention.

Module 8 is the culmination. Here you integrate everything into a production-grade pipeline that takes your AI application from a commit all the way to production, fully automated. It's not "pasting the previous workflows into a single file." It's designing a cohesive system where each stage knows what to do if the previous stage failed, where data flows between jobs through outputs and artifacts, and where a post-deploy failure triggers an automatic rollback without anyone having to intervene.

Integration is a skill in itself. Knowing how to build Docker, knowing how to run tests, and knowing how to deploy are individual abilities. Connecting them into a pipeline where the Docker build depends on the tests passing, where the deploy uses the image Docker built, and where a failure in the post-deploy smoke test triggers a rollback that uses the previous image — that's integration engineering. And it's exactly what you'll build here.

Context: Where are we in the guide?

You've completed 7 modules. You have all the pieces:

• ✅ Module 1: Workflows, jobs, steps, YAML, triggers
• ✅ Module 2: pytest, matrix testing, caching, test reports
• ✅ Module 3: Prompt regression, cost estimation, quality gates
• ✅ Module 4: GitHub Secrets, environments, OIDC
• ✅ Module 5: Docker build, push, layer caching, GHCR
• ✅ Module 6: Staging, production, approval gates, rollback strategies
• ✅ Module 7: Notifications, scheduled workflows, reusable patterns

This is the final module. What you build here is portfolio-worthy.

Why an integration module

In the industry, most CI/CD pipelines don't fail from a lack of technical knowledge. They fail from integration problems:

• One job produces an output, but the next job consumes it under the wrong name
• The rollback works in staging but fails in production because the environment has different variables
• Notifications arrive for CI failures, but not for deploy failures because the notify job doesn't have if: always()
• Cost monitoring runs, but the comparison artifact can't be found because its retention expired

These bugs don't show up when you test each piece separately. They only show up when everything runs together. That's why this module exists: to force you to find and solve the integration bugs before they reach production.

An AI engineer's value isn't just knowing how to configure a workflow. It's knowing how to design a system of workflows that works as a coherent, resilient, and observable whole.

The module's objective

By completing this module you'll be able to:

• ✅ Design the architecture of a production-grade pipeline: stages, dependencies, the data flow between jobs
• ✅ Implement the complete pipeline: lint → test → AI checks → Docker build → push → deploy staging → smoke tests → approve → deploy production
• ✅ Configure automatic rollback: if the deploy fails or the smoke tests fail, redeploy the previous version
• ✅ Implement cost monitoring: tracking the estimated cost per deployment, alerts if the cost rises
• ✅ Create pipeline documentation: a pipeline README, a troubleshooting runbook, a flow diagram
• ✅ Implement post-deployment validation: a health check + a prompt test after the deploy
• ✅ Run the pipeline end-to-end at least once successfully and once with a failure + rollback

Prerequisites

Required knowledge

• ✅ Modules 1-7 completed: You have all the individual pieces
• ✅ The Module 7 pipeline working: You have CI with notifications, health checks, and reusable patterns
• ✅ The Module 5 Docker workflow: You know how to build and push images
• ✅ The Module 6 deployment: You understand staging, production, approval gates

Required tools

• ✅ A GitHub repo with Actions enabled
• ✅ Configured secrets: OPENAI_API_KEY, SLACK_WEBHOOK_URL
• ✅ Accessible GHCR (GitHub Container Registry)
• ✅ Configured GitHub Environments: staging, production

# Verify Docker
docker --version

# Verify that you can push to GHCR
echo $GITHUB_TOKEN | docker login ghcr.io -u your-username --password-stdin

# Verify the secrets (in the GitHub UI)
# Settings → Secrets → OPENAI_API_KEY ✅
# Settings → Secrets → SLACK_WEBHOOK_URL ✅

# Verify the environments (in the GitHub UI)
# Settings → Environments → staging ✅
# Settings → Environments → production (with required reviewers) ✅

The module's content

These are the 8 lessons that make up this module:

Lesson 02: Pipeline Architecture

How to connect stages, pass data between jobs (outputs, artifacts), handle dependencies. The complete pipeline diagram and the architecture decisions.

The main skill: Designing the data flow between jobs — knowing when to use outputs (a short string, immediate) vs artifacts (a file, with retention).

Lesson 03: Full Pipeline Implementation

The complete pipeline in a single YAML: lint → test → AI checks → Docker build → push → deploy staging → smoke tests → approve → deploy production. The real implementation, not a sketch.

The main skill: Writing and debugging a ~200-line YAML with multiple interdependent jobs.

Lesson 04: Automatic Rollback

If the deploy fails or the post-deploy smoke tests fail, the pipeline automatically redeploys the previous version. Edge cases: the first deploy (there's no previous version), a rollback that also fails.

The main skill: Implementing continue-on-error + conditionals to create a recovery flow inside a job.

Lesson 05: Cost Monitoring in the Pipeline

Tracking the estimated cost of prompts per deployment, comparing against the previous deployment, alerting if the cost rises above a threshold. An artifact with the cost history.

The main skill: Integrating analysis scripts (Python) into the pipeline and using artifacts to create a data history.

Lesson 06: Pipeline Documentation

The pipeline README, a troubleshooting runbook, an ASCII flow diagram. Documentation is as important as the code — without it, the pipeline becomes a black box.

The main skill: Creating operational documentation that reduces the bus factor and speeds up troubleshooting.

Lesson 07: Post-Deployment Validation

Smoke tests in production: a health check + a basic prompt test. What to do if the validation fails (trigger an automatic rollback). The difference between "the deploy completed" and "the application works correctly."

The main skill: Designing validations for AI systems that balance reliability (retries, timeouts) with feedback speed.

Lesson 08: Final Project — Production AI Pipeline

THE deliverable. A complete commit-to-production pipeline that works end-to-end. It has to run at least once successfully (the happy path) and once with a failure + rollback (the error path). Portfolio-worthy.

The main skill: Integrating every component, testing end-to-end on both paths (success and failure), and documenting the result.

The complete pipeline: An overview

Commit
  │
  ├─► CI Stage
  │   ├── Lint (ruff)
  │   ├── Test (pytest, matrix)
  │   └── AI Checks (prompt regression, cost estimation)
  │
  ├─► Build Stage
  │   ├── Docker build (layer caching)
  │   ├── Image tagging (SHA + latest)
  │   ├── Security scan (trivy)
  │   └── Push to GHCR
  │
  ├─► Deploy Staging
  │   ├── Pull image
  │   ├── Deploy to staging environment
  │   └── Smoke tests (health + prompt)
  │
  ├─► Approval Gate
  │   └── Manual approval from reviewer
  │
  ├─► Deploy Production
  │   ├── Save current production tag (for rollback)
  │   ├── Pull new image
  │   ├── Deploy to production environment
  │   └── Post-deploy validation
  │
  ├─► Rollback (if validation fails)
  │   ├── Redeploy previous tag
  │   └── Notify: rollback executed
  │
  └─► Notifications
      ├── Failure at any stage → Slack alert
      ├── Deploy to production success → Slack confirmation
      └── Rollback executed → Slack alert

Every previous module contributed a piece:

What's different in this module

The previous modules taught you isolated features. This module teaches you three integration skills:

1. The data flow between jobs

jobs:
  build:
    outputs:
      image-tag: ${{ steps.meta.outputs.tag }}
    steps:
      - id: meta
        run: echo "tag=ghcr.io/org/app:abc123" >> $GITHUB_OUTPUT

  deploy:
    needs: build
    steps:
      - run: docker pull ${{ needs.build.outputs.image-tag }}

The Docker image tag travels from the build job to the deploy job through outputs. Without this, there's no way for deploy to know which image to use.

2. Error recovery

  deploy:
    steps:
      - name: Save current tag for rollback
        run: echo "current_tag=abc123" >> $GITHUB_OUTPUT

      - name: Deploy new version
        run: ./deploy.sh ${{ needs.build.outputs.image-tag }}

  rollback:
    needs: deploy
    if: failure()
    steps:
      - name: Redeploy previous version
        run: ./deploy.sh ${{ needs.deploy.outputs.current-tag }}

If deploy fails, the rollback job activates and redeploys the previous version.

3. Notification orchestration

Not a generic "CI failed" notification, but specific notifications per stage:

• ✅ "AI checks failed — a possible regression in the prompt"
• ✅ "The Docker build failed — check the Dockerfile"
• ✅ "The deploy to production succeeded — version abc123 is live"
• ✅ "A rollback was executed — the previous version was restored"

The approach: Incremental, not big bang

You're not going to write a 300-line pipeline in one shot. The approach is incremental:

1. Lesson 02: Understand the architecture and the data flow
2. Lesson 03: Implement the complete pipeline (the big YAML)
3. Lesson 04: Add automatic rollback
4. Lesson 05: Add cost monitoring
5. Lesson 06: Document everything
6. Lesson 07: Add post-deployment validation
7. Lesson 08: Integrate, test end-to-end, validate

Each lesson adds a layer to the pipeline. By the end of lesson 03 you already have a functional pipeline. Lessons 04-07 make it production-grade. Lesson 08 is the final validation.

An analogy: a symphony orchestra

Think of the previous modules as individual musicians practicing their instrument:

• Module 1-2: The violinist practices scales (lint, test)
• Module 3: The percussionist learns complex rhythms (AI checks)
• Module 4: The sound engineer prepares the equipment (secrets, security)
• Module 5: The set designer builds the stage (Docker, containers)
• Module 6: Each section rehearses together (deploy, basic rollback)
• Module 7: The conductor establishes the cues (notifications, scheduling, patterns)

Module 8 is the concert. Everyone plays together, following the same score. The conductor coordinates entrances, exits, and recoveries. If a musician makes a mistake (the deploy fails), the conductor has a plan: go back to the last stable bar (the rollback).

The difference between an individual rehearsal and a concert is the same difference between an isolated job and an integrated pipeline. Coordination is a skill you only learn by doing.

The recommended technical setup

Before starting, make sure you have:

Tools

# Verify that you have everything installed:
python --version    # Python 3.10+
docker --version    # Docker 20+
gh --version        # GitHub CLI 2.0+
git --version       # Git 2.30+

If any tool is missing, install it before continuing. The module assumes everything is ready.

The estimated time

The prepared repo

Your repo should have (from Module 7):

.github/
├── actions/
│   └── setup-ai-project/
│       └── action.yml          # Composite action
├── workflows/
│   ├── reusable-ai-ci.yml     # Reusable workflow
│   └── ci.yml                 # Main pipeline
scripts/
├── pipeline_metrics.py         # Health metrics
├── prompt_regression.py        # Prompt checks
└── cost_estimation.py          # Cost estimation (you'll create it here)
tests/
├── test_main.py
└── conftest.py
src/
├── main.py
└── prompts.json
Dockerfile
requirements.txt

The configured secrets

In your repo → Settings → Secrets and variables → Actions:

• OPENAI_API_KEY — for the AI checks
• SLACK_WEBHOOK_URL — for the notifications
• DEPLOY_SSH_KEY — for the deploy (simulated is fine)

The configured environments

In your repo → Settings → Environments:

• staging — with no protection
• production — with required reviewers

If you don't have the environments configured, lessons 03-07 guide you through creating them.

What this module does NOT cover

• ❌ Real cloud infrastructure: The deploy simulates production with Docker Compose. The real cloud (AWS, GCP) is guide #17.
• ❌ Runtime monitoring: Monitoring the app in production is guide #18. This module monitors the pipeline, not the app.
• ❌ Multi-cloud deployment: You only deploy to one target. Multi-region is an advanced infrastructure topic.
• ❌ Kubernetes: The deploy is Docker Compose. K8s deployments are a topic for the Cloud guide.
• ❌ Database migrations: The pipeline doesn't include migrations. That's an application architecture topic.

Evidence of success

By the end of this module, you should be able to:

• Explain how data flows between jobs (outputs and artifacts)
• Have a complete pipeline YAML that runs lint → test → AI checks → Docker → deploy
• Run the pipeline successfully end-to-end (the happy path)
• Provoke a failure and see the automatic rollback in action
• Have a pipeline README with a flow diagram and troubleshooting
• Have an artifact with the deployment's cost report
• Receive notifications in Slack for a deploy success and a rollback

If you check every box → you completed the guide.

After this module

By the end, you have a production-grade CI/CD pipeline for AI systems. The next steps in the AI Engineering Path:

• ✅ Guide #17 (Deployment & Cloud Infrastructure): Take your pipeline to AWS/GCP with infrastructure as code
• ✅ Guide #18 (Monitoring & Observability): Monitor your AI application in production — latency, cost, quality, errors

Your pipeline from this guide integrates directly with both: #17 replaces the simulated deploy with a real deploy to the cloud, and #18 adds monitoring of the app (not just of the pipeline).

Additional resources

1. GitHub Actions — Workflow Syntax - The complete YAML reference
2. GitHub Actions — Job Outputs - How to pass data between jobs
3. GitHub Environments - Configuring environments
4. Docker Compose Deploy - Docker Compose in production
5. GitHub Actions — Deployment Strategies - Deployment strategies
6. DORA Metrics - The industry-standard metrics for DevOps performance