2. Pipeline Architecture

Overview

Before writing a single line of YAML, you need to understand the complete pipeline's architecture. How the stages connect, how the data flows between jobs, what depends on what, and what happens when something fails halfway through. Without this architectural view, you end up with a long YAML that "works" but that you can't debug, extend, or explain.

A production-grade pipeline isn't a list of sequential steps. It's a directed graph of jobs with dependencies, outputs that flow from one job to another, conditions that determine what runs and what gets skipped, and error handlers that know how to recover. Designing this graph correctly is the difference between a fragile pipeline and a resilient one.

Connection with the final pipeline: The architecture you design in this lesson is exactly the one you'll implement in Lesson 03 (Full Pipeline Implementation) and that will become your final project.

The complete diagram

                    push to main
                        │
            ┌───────────┼───────────┐
            ▼           ▼           ▼
         ┌──────┐  ┌────────┐  ┌──────────┐
         │ Lint │  │  Test  │  │ AI Checks│
         └──┬───┘  └───┬────┘  └────┬─────┘
            │          │            │
            └──────────┼────────────┘
                       │
                       ▼
               ┌──────────────┐
               │ Docker Build │
               │  & Push      │
               └──────┬───────┘
                      │
                      ▼  outputs: image-tag
               ┌──────────────┐
               │Deploy Staging│
               └──────┬───────┘
                      │
                      ▼
               ┌──────────────┐
               │ Smoke Tests  │
               └──────┬───────┘
                      │
                      ▼
               ┌──────────────┐
               │Approval Gate │ (manual)
               └──────┬───────┘
                      │
                      ▼
               ┌──────────────┐
               │Deploy Prod   │──── failure ──► Rollback
               └──────┬───────┘                    │
                      │                            │
                      ▼                            ▼
               ┌──────────────┐           ┌──────────────┐
               │Post-Deploy   │           │Notify:       │
               │Validation    │           │Rollback      │
               └──────┬───────┘           └──────────────┘
                      │
                      ▼
               ┌──────────────┐
               │Notify:       │
               │Success       │
               └──────────────┘

The stages and their responsibilities

Stage 1: CI Checks (in parallel)

jobs:
  lint:
    runs-on: ubuntu-latest
    # No dependencies → it runs immediately

  test:
    runs-on: ubuntu-latest
    # No dependencies → it runs in parallel with lint

  ai-checks:
    runs-on: ubuntu-latest
    # No dependencies → it runs in parallel with lint and test

Parallelism: The 3 jobs run at the same time. If your pipeline has 3 minutes of tests and 2 minutes of AI checks, Stage 1 takes 3 minutes (not 5).

A design decision: Why separate jobs and not a single job with sequential steps?

For a production-grade pipeline, the visibility and the ability to re-run justify the extra cost of separate runners.

Stage 2: Docker Build & Push

  docker:
    needs: [lint, test, ai-checks]
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'

Dependencies: It only runs after the 3 CI checks pass. If any of them fails, the Docker build doesn't run.

The conditional: It only runs on a push to main. On PRs, the CI checks are enough — you don't need to build Docker for every PR.

The critical output: This job produces the image-tag that the later stages need.

Stage 3: Deploy Staging

  deploy-staging:
    needs: docker
    environment:
      name: staging

Dependencies: It needs the Docker image tag from the previous stage.

Environment: It uses GitHub's staging environment, which can have its own secrets and variables.

Stage 4: Smoke Tests

  smoke-tests:
    needs: deploy-staging

It verifies that the app deployed to staging works: a health check + a basic prompt test.

Stage 5: Approval Gate

  approve:
    needs: smoke-tests
    environment:
      name: production

The production environment has "required reviewers" configured. The pipeline pauses here until a reviewer approves.

Stage 6: Deploy Production + Validation

  deploy-production:
    needs: approve

  rollback:
    needs: deploy-production
    if: failure()

A deploy to production with post-deploy validation. If the validation fails, the rollback job activates.

The data flow between jobs

The problem: Jobs don't share state

Each job runs on a different runner. There's no shared filesystem, no shared variables, nothing. If the docker job builds an image and tags it as sha-abc123, the deploy-staging job doesn't know what that tag is unless you pass it explicitly.

Solution 1: Job outputs

jobs:
  docker:
    runs-on: ubuntu-latest
    outputs:
      image-tag: ${{ steps.meta.outputs.tag }}
      image-digest: ${{ steps.push.outputs.digest }}
    steps:
      - name: Generate image metadata
        id: meta
        run: |
          TAG="ghcr.io/${{ github.repository }}:sha-${GITHUB_SHA::7}"
          echo "tag=$TAG" >> $GITHUB_OUTPUT

      - name: Push image
        id: push
        run: |
          docker push $TAG
          DIGEST=$(docker inspect --format='{{.RepoDigests}}' $TAG)
          echo "digest=$DIGEST" >> $GITHUB_OUTPUT

  deploy-staging:
    needs: docker
    steps:
      - name: Pull and deploy
        run: |
          docker pull ${{ needs.docker.outputs.image-tag }}
          IMAGE_TAG=${{ needs.docker.outputs.image-tag }} docker compose up -d

The limitations of outputs:

• Strings only (no files, no complex objects)
• A maximum of ~1MB per output
• They're lost if the job fails before writing them

Solution 2: Artifacts

For larger data (reports, configuration files):

jobs:
  build:
    steps:
      - name: Generate cost report
        run: python scripts/cost_report.py > cost-report.json

      - name: Upload artifact
        uses: actions/upload-artifact@v4
        with:
          name: cost-report
          path: cost-report.json

  analyze:
    needs: build
    steps:
      - name: Download artifact
        uses: actions/download-artifact@v4
        with:
          name: cost-report

      - name: Analyze costs
        run: cat cost-report.json | python scripts/analyze_costs.py

Outputs vs Artifacts:

When to use each one

Is it a short string (a tag, a URL, a flag)?
  → An output

Is it a file (a report, a log, an image)?
  → An artifact

Do you need to access it after the workflow?
  → An artifact (with retention)

Do you only need it in the next job?
  → An output (simpler)

Handling dependencies

A simple needs

  deploy:
    needs: docker

deploy waits for docker to finish successfully. If docker fails, deploy gets skipped.

A multiple needs

  docker:
    needs: [lint, test, ai-checks]

docker waits for ALL THREE to finish successfully.

needs with a condition

  notify:
    needs: [lint, test, ai-checks]
    if: always()

notify always runs, even if one of the jobs failed. Without if: always(), it wouldn't run in the case of a failure.

The rollback pattern

  deploy-production:
    needs: approve
    steps:
      - id: deploy
        run: ./deploy.sh
      - id: validate
        run: ./validate.sh

  rollback:
    needs: deploy-production
    if: failure()
    steps:
      - run: ./rollback.sh

if: failure() activates when the previous job (deploy-production) failed. This is the fundamental pattern of automatic rollback.

Architecture decisions

One workflow or several?

The recommendation: One main workflow for the core pipeline (CI → build → deploy). Separate workflows for auxiliary operations (nightly health, weekly report).

Sequential or parallel CI?

# Sequential (slower, simpler)
jobs:
  ci:
    steps:
      - run: ruff check .
      - run: pytest tests/
      - run: python scripts/prompt_regression.py

# Parallel (faster, more complex)
jobs:
  lint: ...
  test: ...
  ai-checks: ...

The recommendation: Parallel for the CI checks (it saves time). Sequential for the deploy stages (the order matters).

A simulated or a real deploy?

In this guide we use Docker Compose as the deploy target. It's not AWS, it's not GCP — it's a local simulation that demonstrates the patterns without requiring cloud infrastructure.

A real pipeline:
  Docker build → Push to GHCR → Deploy to EC2 via SSH

This guide's pipeline:
  Docker build → Push to GHCR → Deploy via docker compose up

The patterns are identical. Only the target changes. When you migrate to the cloud (guide #17), you replace the deploy step without changing the pipeline's architecture.

Comparisons

A linear pipeline vs a pipeline with branches

A pipeline with rollback vs without rollback

Troubleshooting

"A job's output arrives empty at the next one"

Cause: The step that writes the output didn't run or had an error.

Solution: Verify that the step has an id and writes correctly to $GITHUB_OUTPUT:

- id: meta
  run: echo "tag=my-value" >> $GITHUB_OUTPUT

And that the job declares the output:

jobs:
  build:
    outputs:
      tag: ${{ steps.meta.outputs.tag }}

"The deploy job gets skipped even though the CI checks passed"

Cause: The job has an if condition that isn't met.

Solution: Verify the conditions. A common error:

# ❌ This gets skipped on PRs
if: github.event_name == 'push'

# ✅ This runs on push and on PR
if: github.event_name == 'push' || github.event_name == 'pull_request'

"The artifact can't be found in the next job"

Cause: The artifact's name doesn't match between upload and download.

Solution: Verify that the name is exactly the same:

# Upload
uses: actions/upload-artifact@v4
with:
  name: cost-report    # ← This name

# Download
uses: actions/download-artifact@v4
with:
  name: cost-report    # ← It has to match exactly

"The rollback doesn't activate when the deploy fails"

Cause: The rollback job doesn't have if: failure() or doesn't depend on the right job.

Solution:

rollback:
  needs: deploy-production    # It depends on the job that can fail
  if: failure()                # It activates when deploy-production fails

Exercises

Exercise 1: Design the data flow

Draw the data flow diagram for your pipeline: what outputs each job produces and which job consumes them.

lint       → (no outputs)
test       → outputs: test-result, coverage
ai-checks  → outputs: regression-status, cost-estimate

docker     → outputs: image-tag, image-digest
              (consumes: nothing from the previous jobs, just needs for ordering)

deploy-staging → outputs: staging-url
                  (consumes: docker.outputs.image-tag)

smoke-tests → outputs: health-status
               (consumes: deploy-staging.outputs.staging-url)

deploy-prod → outputs: previous-tag, deploy-status
               (consumes: docker.outputs.image-tag)

rollback   → (consumes: deploy-prod.outputs.previous-tag)

notify     → (consumes: the results of all of them via needs.*.result)

Exercise 2: Write the needs graph

For the complete pipeline, write out each job's dependencies.

jobs:
  lint:                                    # No needs (it runs immediately)
  test:                                    # No needs (in parallel with lint)
  ai-checks:                              # No needs (in parallel with lint and test)
  docker:      needs: [lint, test, ai-checks]  # It waits for the complete CI
  deploy-staging: needs: docker            # It waits for the Docker build
  smoke-tests: needs: deploy-staging       # It waits for the deploy to staging
  approve:     needs: smoke-tests          # It waits for the smoke tests
  deploy-prod: needs: approve             # It waits for the approval
  rollback:    needs: deploy-prod          # It activates if the deploy fails
               if: failure()
  notify:      needs: [deploy-prod, rollback]  # It always runs at the end
               if: always()

Exercise 3: Output vs Artifact

For each piece of data, decide whether you'd use an output or an artifact:

1. The Docker image's tag (ghcr.io/org/app:sha-abc123)
2. A JSON cost estimation report (50KB)
3. A boolean flag: did the AI checks pass?
4. pytest's complete logs (2MB)

Architecture best practices

1. Naming conventions

Use clear and consistent names for jobs:

jobs:
  lint:          # ✅ Clear
  test:          # ✅ Clear
  ai-checks:    # ✅ Descriptive, with a prefix
  docker:        # ✅ Clear
  deploy-staging:    # ✅ It includes the target
  deploy-production: # ✅ It includes the target
  smoke-tests:       # ✅ Descriptive
  notify:            # ✅ Clear

Avoid generic names like job1, build-and-test, or deployment.

2. A timeout on every job

Always define timeout-minutes. Without a timeout, a hung job can eat your Actions budget:

jobs:
  lint:
    timeout-minutes: 5
  test:
    timeout-minutes: 10
  ai-checks:
    timeout-minutes: 15   # External APIs need more margin
  docker:
    timeout-minutes: 15
  deploy-production:
    timeout-minutes: 10

3. The principle of least privilege for secrets

Pass secrets only to the jobs that need them:

  ai-checks:
    env:
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}  # Only here

  deploy-production:
    env:
      SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}  # Only here

Don't use secrets: inherit unless it's a reusable workflow that genuinely needs access to all of them.

4. A clear separation: CI vs CD

The CI jobs (lint, test, ai-checks) run on every trigger (push, PR). The CD jobs (docker, deploy) only run on main:

  docker:
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'

This prevents accidental deploys from PRs.

Summary

• ✅ The pipeline has 6 stages: CI (in parallel) → Docker → Staging → Smoke Tests → Approval → Production
• ✅ The data flow with outputs: short strings (tags, flags) pass between jobs via $GITHUB_OUTPUT
• ✅ The data flow with artifacts: files (reports, logs) pass between jobs via upload/download-artifact
• ✅ Dependencies with needs: they control the execution order and the propagation of failures
• ✅ if: failure() activates the rollback when the deploy fails
• ✅ if: always() guarantees that notifications run no matter the result
• ✅ Parallel CI, sequential deploy: it maximizes speed in the checks and keeps the order in the deployment
• ✅ One main workflow + auxiliary workflows: a balance between visibility and maintainability

Additional resources

1. GitHub Actions — Job Outputs - How to pass data between jobs
2. GitHub Actions — Artifacts - Uploading and downloading artifacts
3. GitHub Actions — needs - Dependencies between jobs
4. GitHub Actions — Status Check Functions - failure(), always(), success()
5. GitHub Actions — Workflow Visualization - Visualizing the pipeline's graph
6. Pipeline Architecture Patterns - Patterns for workflows with dependencies