6. Pipeline Documentation

Overview

Your pipeline has 12 jobs, 50+ steps, secrets, environments, approval gates, a rollback, and notifications. If a new developer joins the team and asks "how does our pipeline work?", is the answer "read the 300-line YAML"? Without documentation, the pipeline becomes a black box that only the person who created it understands.

Pipeline documentation isn't a nice-to-have — it's a requirement for maintainability. It includes three pieces: a README that explains each stage of the pipeline, a runbook for troubleshooting common failures, and a flow diagram that visualizes the architecture. These documents are as important as the pipeline's YAML: the YAML says what the pipeline does, the documentation says why and how to debug it.

Connection with the final pipeline: The documentation you create here — the pipeline README, the rollback runbook, the flow diagram — is part of the final project's deliverable in Lesson 08.

The pipeline README

The recommended structure

# Production AI Pipeline

## Overview

A complete CI/CD pipeline for [project-name]. It automates from
the commit all the way to production with AI quality checks, a Docker build,
a staged deployment, and automatic rollback.

## Architecture

[The flow diagram]

## Stages

### 1. CI Checks (parallel)
- **Lint:** ruff check + format check
- **Test:** pytest with coverage
- **AI Checks:** prompt regression + cost estimation

### 2. Docker Build & Push
- A build with layer caching
- A push to GHCR with tags: SHA + latest
- It only runs on a push to main

### 3. Deploy Staging
- It deploys automatically to the staging environment
- No approval required

### 4. Smoke Tests
- A health check of the application
- A basic prompt test against the OpenAI API

### 5. Approval Gate
- It requires manual approval from [reviewer]
- Review staging before approving

### 6. Deploy Production
- It saves the previous version (for the rollback)
- It deploys the new version
- Post-deploy validation (health + prompt)
- An automatic rollback if the validation fails

### 7. Notifications
- Failure → a Slack alert
- Production deploy → a Slack confirmation
- Rollback → a Slack alert

## Secrets Required

| Secret | Description | Where to get it |
|--------|-------------|-----------------|
| `OPENAI_API_KEY` | The OpenAI API key | platform.openai.com |
| `SLACK_WEBHOOK_URL` | The Slack webhook | api.slack.com/apps |
| `DEPLOY_SSH_KEY` | The SSH key for deployment | Generated by the team |
| `GITHUB_TOKEN` | Auto-generated | GitHub (automatic) |

## Environments

| Environment | URL | Approval | Secrets |
|-------------|-----|----------|---------|
| staging | staging.your-app.com | No | DEPLOY_SSH_KEY, STAGING_HOST |
| production | your-app.com | Yes (1 reviewer) | DEPLOY_SSH_KEY, PRODUCTION_HOST |

## Workflows

| File | Trigger | Description |
|---------|---------|-------------|
| `production-pipeline.yml` | push main, PR | The complete pipeline |
| `nightly-health.yml` | cron 4am UTC | The nightly health check |
| `weekly-report.yml` | cron Monday 9am | The weekly report |

## Cost

- GitHub Actions: ~$0.04 per run (8 min × $0.005/min Linux runner)
- OpenAI API: ~$0.05 per run (the AI checks)
- Total: ~$0.09 per run, ~$1.80/week (20 PRs)

The ASCII flow diagram

The ASCII diagram is fundamental because it renders anywhere: a GitHub README, a terminal, Slack, email. It doesn't depend on external tools.

The complete diagram

                    ┌─────────────────────┐
                    │   push to main      │
                    └──────────┬──────────┘
                               │
              ┌────────────────┼────────────────┐
              ▼                ▼                 ▼
        ┌──────────┐   ┌──────────┐   ┌───────────────┐
        │   Lint   │   │   Test   │   │   AI Checks   │
        │  (ruff)  │   │ (pytest) │   │ (regression,  │
        │  ~30s    │   │  ~2min   │   │  cost est.)   │
        └────┬─────┘   └────┬─────┘   │  ~3min        │
             │              │         └───────┬───────┘
             └──────────────┼─────────────────┘
                            │
                  All pass? │
                     YES ───┤
                            ▼
                ┌───────────────────────┐
                │   Docker Build & Push │
                │   (GHCR, layer cache) │
                │   ~3min               │
                └───────────┬───────────┘
                            │
                            ▼  image-tag
                ┌───────────────────────┐
                │    Deploy Staging     │
                │   (docker compose)    │
                └───────────┬───────────┘
                            │
                            ▼
                ┌───────────────────────┐
                │     Smoke Tests       │
                │  (health + prompt)    │
                └───────────┬───────────┘
                            │
                            ▼
                ┌───────────────────────┐
                │   Approval Gate ⏸️    │
                │  (manual approval)    │
                └───────────┬───────────┘
                            │ approved
                            ▼
                ┌───────────────────────┐
                │  Deploy Production    │──── fail ──► Rollback
                │  (save prev, deploy,  │              │
                │   validate)           │              ▼
                └───────────┬───────────┘        Notify: ⚠️
                            │ success              Rollback
                            ▼
                    Notify: 🚀 Success

The simplified diagram (for Slack/chat)

CI → Docker → Staging → Tests → Approve → Production
                                              │
                                         fail → Rollback

The troubleshooting runbook

A runbook is a list of common problems with specific steps to resolve them. It's the first place a developer looks when something fails.

The runbook's structure

# Pipeline Troubleshooting Runbook

## Quick Reference

| Symptom | Likely cause | The quick fix |
|---------|---------------|-----------------|
| Lint fails | A formatting issue | `ruff format src/` |
| Tests fail | A code bug or a flaky test | Check the logs, re-run |
| AI checks fail | A model update or rate limiting | Check the baselines |
| The Docker build fails | A Dockerfile error or the cache | Check the Dockerfile |
| The deploy fails | An SSH/network issue | Verify the SSH key |
| The health check fails | An app crash or a port issue | Check the logs |
| A rollback ran | The deploy validation failed | Check the post-deploy |

---

## The Problems in Detail

### 1. Lint fails: "ruff check found errors"

**Diagnosis:**
```bash
ruff check src/ --diff
```

**Solution:**
```bash
ruff check src/ --fix
ruff format src/
git add . && git commit -m "fix: lint errors"
```

**Prevention:** Configure a pre-commit hook with ruff.

---

### 2. Tests fail: "FAILED tests/test_main.py::test_X"

**Diagnosis:**
1. Open the workflow run in GitHub Actions
2. Click on the "Test" job
3. Read the traceback of the test that failed

**If it's a flaky test (an intermittent failure):**
- Re-run the job
- If it passes on the retry, it's flaky
- Flag the test for investigation

**If it's a real failure:**
```bash
pytest tests/test_X.py -v --tb=long
```

---

### 3. The AI Checks fail: "Prompt regression detected"

**Diagnosis:**
- Did you change a prompt? → Update the baselines
- You didn't change anything? → A possible model update

**The solution if it's a model update:**
```bash
python scripts/prompt_regression.py --mode generate-baseline
git add baselines/
git commit -m "chore: update baselines after model update"
```

**The solution if the cost exceeds the threshold:**
```bash
python scripts/cost_estimation.py --prompts prompts.json
```
Check which prompt changed its model or its token count.

---

### 4. The Docker build fails

**Common causes:**
- The layer cache got invalidated → A complete rebuild (normal, ~5 min)
- `requirements.txt` has an incompatible package → Fix the version
- A Dockerfile syntax error → Check the Dockerfile

**Diagnosis:**
```bash
docker build -t test . 2>&1 | tail -20
```

---

### 5. The deploy fails: "SSH connection refused"

**Diagnosis:**
1. Verify that the server is up
2. Verify that the SSH key secret is correct
3. Verify the hostname/IP

**Solution:**
```bash
# Test the SSH connection
ssh -i /path/to/key deploy@your-server "echo ok"
```

---

### 6. The health check fails post-deploy

**Diagnosis:**
1. The deploy completed but the app doesn't respond
2. Check the container's logs

```bash
ssh deploy@your-server "cd /app && docker compose logs --tail 50"
```

**Common causes:**
- A port mismatch between the Dockerfile and compose
- A missing environment variable
- A database connection error
- An expired API key

---

### 7. A rollback ran automatically

**This isn't an error — it's the system working correctly.**

**The post-rollback steps:**
1. Verify that production is stable (the rollback succeeded)
2. Investigate why the deploy failed
3. Fix the issue in a new PR
4. The next deploy will be automatic

When to update the documentation

Documentation goes stale fast if you don't maintain it. These are the rules:

Automating the verification

- name: Verify pipeline docs are up to date
  run: |
    WORKFLOWS=$(ls .github/workflows/*.yml | wc -l)
    DOCUMENTED=$(grep -c "\.yml" .github/PIPELINE.md || echo "0")
    if [ "$WORKFLOWS" != "$DOCUMENTED" ]; then
      echo "::warning::Pipeline docs may be outdated. $WORKFLOWS workflows, $DOCUMENTED documented."
    fi

Comparisons

With documentation vs without documentation

A README vs a Wiki vs inline comments

Troubleshooting

"The documentation goes stale constantly"

Cause: There's no process for keeping it up to date.

Solution: Add a checklist to your PR template:

## PR Checklist
- [ ] Tests pass
- [ ] Lint pass
- [ ] Pipeline documentation updated (if pipeline changed)

"I don't know what to include in the runbook"

Cause: You haven't experienced enough failures.

Solution: Start with the problems you've already hit. Every time you debug a failure, add the solution to the runbook. In 1-2 months you'll have a complete runbook.

"The ASCII diagram is hard to maintain"

Cause: ASCII art is manual and tedious.

Solution: Use tools that generate ASCII:

• asciiflow.com — A visual editor for ASCII art
• mermaid — If your README renders on GitHub (it supports mermaid)

Exercises

Exercise 1: Write your pipeline's README

Create a .github/PIPELINE.md file with your pipeline's documentation.

Exercise 2: Create a runbook with 5 problems

Document the 5 most common problems you've hit with your pipeline.

### The tests fail from an API timeout

**Symptom:** pytest fails with "TimeoutError" on the tests that call OpenAI.

**Cause:** Rate limiting or a slow OpenAI API.

**Diagnosis:** Check status.openai.com. Re-run the job.

**Solution:** If it's rate limiting, add retry logic. If it's an outage,
wait and re-run.

**Prevention:** Add explicit timeouts and retries with backoff in the tests.

Exercise 3: An ASCII diagram of your pipeline

Create an ASCII diagram that shows your pipeline's flow, including the parallel CI jobs and the rollback path.

        ┌──────┐  ┌──────┐  ┌───────────┐
        │ Lint │  │ Test │  │ AI Checks │
        └──┬───┘  └──┬───┘  └─────┬─────┘
           └─────────┼────────────┘
                     ▼
              ┌──────────────┐
              │ Docker Build │
              └──────┬───────┘
                     ▼
              ┌──────────────┐
              │   Staging    │
              └──────┬───────┘
                     ▼
              ┌──────────────┐
              │ Smoke Tests  │
              └──────┬───────┘
                     ▼
              ┌──────────────┐
              │   Approve    │
              └──────┬───────┘
                     ▼
              ┌──────────────┐
              │  Production  │─── fail ──► Rollback
              └──────┬───────┘
                     ▼
                  Success 🚀

Exercise 4: Automate the docs verification

Create a step in your pipeline that warns if the documentation might be stale.

- name: Check documentation freshness
  run: |
    PIPELINE_MODIFIED=$(git log -1 --format=%ct -- .github/workflows/)
    DOCS_MODIFIED=$(git log -1 --format=%ct -- .github/PIPELINE.md 2>/dev/null || echo "0")

    if [ "$PIPELINE_MODIFIED" -gt "$DOCS_MODIFIED" ]; then
      echo "::warning::Pipeline workflows modified more recently than documentation."
      echo "::warning::Consider updating .github/PIPELINE.md"
    fi

Documentation best practices

1. Write for the developer at 3am

The runbook is going to be read by somebody half asleep, under pressure, during an incident. Make it:

• Scannable: Use tables and bullets, not long paragraphs
• Copy-pasteable: The commands have to work as-is (no generic placeholders)
• Linear: Top to bottom — symptom → diagnosis → solution
• Unambiguous: Not "check the service" but "run curl -sf https://your-app.com/health"

2. Document the "whys," not just the "whats"

<!-- ❌ Only the "what" -->
## Lint
The lint job uses ruff.

<!-- ✅ The "what" + the "why" -->
## Lint
The lint job uses ruff instead of flake8 because:
- It's 10-100x faster (relevant in CI)
- It includes isort built in (one less dependency)
- It supports auto-fix with `--fix`

3. Include a pipeline changelog

## Pipeline Changelog

| Date | Change | Reason |
|-------|--------|-------|
| 2025-03-01 | Added cost monitoring | To detect API cost spikes |
| 2025-02-15 | Migrated to ruff | flake8 was slow in CI |
| 2025-02-01 | Added an approval gate | To prevent accidental deploys |

4. A PR template with a documentation checklist

Add this to .github/PULL_REQUEST_TEMPLATE.md:

## Checklist

- [ ] Tests pass
- [ ] Linter passes
- [ ] If pipeline was modified: `.github/PIPELINE.md` updated
- [ ] If new secret was added: documented in PIPELINE.md secrets section
- [ ] If new environment was added: documented in PIPELINE.md environments section

This turns updating the documentation into a requirement of the development flow, not a task that gets forgotten.

5. Use internal links

Inside .github/PIPELINE.md, link to specific sections:

If the deploy fails, see the [Troubleshooting Runbook](#troubleshooting-runbook).
If you need to do a manual rollback, see [Manual Rollback](#manual-rollback).

6. Document the expected costs

Include a cost section in your pipeline's README:

## Estimated Costs

### GitHub Actions compute
| Workflow | Frequency | Duration | Monthly cost |
|----------|-----------|----------|-------------|
| CI Pipeline | ~100 runs/month | ~5 min | ~5 min × $0.005/min × 100 = $2.50 |
| Nightly Health | 30 runs/month | ~2 min | ~2 min × $0.005/min × 30 = $0.30 |
| Weekly Report | 4 runs/month | ~1 min | ~1 min × $0.005/min × 4 = $0.02 |

### External APIs
| Service | Usage | Monthly estimate |
|---------|-------|-----------------|
| OpenAI (AI checks) | ~50 runs × $0.01/run | ~$0.50 |
| OpenAI (nightly baseline) | 30 runs × $0.02/run | ~$0.60 |

**Total estimated: ~$3.92/month** (within GitHub free tier for Actions)

This prevents surprises when the invoice arrives and helps the team understand the impact of adding more checks.

Summary

• ✅ Pipeline documentation includes three pieces: the README, the runbook, and the flow diagram
• ✅ The README documents the stages, secrets, environments, workflows, and costs
• ✅ The runbook is the first reference for troubleshooting: symptom → cause → solution
• ✅ The ASCII diagram renders anywhere without depending on external tools
• ✅ Documentation reduces the bus factor — any developer can operate the pipeline
• ✅ Update the documentation when you change the pipeline — add a checklist to the PR template
• ✅ Automate the verification with a step that detects if the docs are stale
• ✅ .github/PIPELINE.md is the standard place for pipeline documentation

Additional resources

1. GitHub — About READMEs - Best practices for READMEs
2. asciiflow.com - A visual editor for ASCII diagrams
3. Mermaid Diagrams in GitHub - Native diagrams in GitHub markdown
4. Google SRE Book — Runbooks - The philosophy of runbooks
5. GitHub Actions — Workflow Commands - ::warning:: and other commands
6. Bus Factor - Why documentation reduces the risk