2. Pipeline Monitoring

Overview

Before configuring automatic notifications, you need to know what to monitor. A pipeline that fails once is a bug. A pipeline that fails three times a week in the same job is a pattern. A pipeline that takes 15 minutes today when it took 5 minutes a month ago is a degradation. If you don't analyze the run history, you can't distinguish between a one-off problem and a systemic one.

Pipeline monitoring is the practice of observing, analyzing, and acting on your CI/CD pipeline's execution history. GitHub Actions provides a dashboard with all the necessary information: workflow runs, duration, success/failure rate, detailed logs per step. The problem isn't a lack of data — it's that most developers never check the dashboard until something blows up.

In this capsule you're going to learn to read GitHub Actions' dashboard as a diagnostic tool, identify patterns of recurring failures, and build a script that generates pipeline health metrics using the GitHub API.

Connection with the final pipeline: In the capstone pipeline (Module 8), the monitoring you learn here gets used to verify that every deployment succeeded and to detect post-deploy degradation.

GitHub Actions' dashboard

Accessing the dashboard

Your repository → The "Actions" tab → The main view

The main view shows:

The available filters

# Filter by status
?query=is%3Afailure     → Failures only
?query=is%3Asuccess     → Successes only

# Filter by branch
?query=branch%3Amain    → Runs on main only

# Filter by event
?query=event%3Apush     → Push triggers only
?query=event%3Aschedule → Scheduled runs only

# Combinations
?query=is%3Afailure+branch%3Amain  → Failures on main

You can apply these filters directly in the URL or use the search bar in the UI.

The anatomy of a workflow run

When you click on a specific run, you see:

Workflow Run: CI Pipeline #142
├── Status: ❌ Failure
├── Branch: feature/update-prompts
├── Trigger: pull_request
├── Duration: 4m 32s
├── Started: 2026-03-07 14:23:05 UTC
│
├── Jobs:
│   ├── lint         ✅ (22s)
│   ├── test         ✅ (1m 45s)
│   ├── ai-checks    ❌ (2m 15s)    ← The failure
│   └── docker-build ⚪ (skipped)   ← It didn't run because ai-checks failed
│
└── Re-run options:
    ├── Re-run all jobs
    └── Re-run failed jobs

What matters here isn't just seeing that it failed, but understanding the cascade: ai-checks failed → docker-build got skipped because it depended on ai-checks → the whole run gets reported as a failure. If you only look at the run's status, you don't know where the problem is. If you look at the individual jobs, you immediately see that ai-checks is the culprit.

Identifying failure patterns

Pattern 1: The same job fails repeatedly

Run #142: ai-checks ❌
Run #140: ai-checks ❌
Run #138: ai-checks ❌
Run #135: ai-checks ✅ (5 days ago)

The question: What changed 5 days ago? The possibilities:

• ✅ A PR that modified the system prompt
• ✅ OpenAI updated gpt-4o-mini and the baselines no longer match
• ✅ The OPENAI_API_KEY secret expired or ran out of credit
• ✅ Rate limiting because another service also uses that API key

Pattern 2: Intermittent (flaky) failures

Run #142: test ❌
Run #141: test ✅
Run #140: test ❌
Run #139: test ✅
Run #138: test ❌

A test that sometimes passes and sometimes fails is a flaky test. In AI systems, flaky tests are more common because:

• ✅ The LLM's non-determinism: Even with temperature=0, there are minimal variations between responses
• ✅ Rate limits: If you run many tests that call the API, some can fail from rate limiting
• ✅ Timeouts: Tests that call external APIs can take longer than expected during high-traffic hours
• ✅ Network issues: GitHub's runners are in the cloud — sometimes the connection to the LLM's API fails

Pattern 3: A gradual degradation in duration

Run #142: 8m 15s   ← Slow
Run #130: 7m 02s
Run #120: 5m 45s
Run #110: 4m 30s
Run #100: 3m 15s   ← Normal

The pipeline got slower gradually. Common causes:

• ✅ A cache miss: The dependency cache expired and every run downloads everything from scratch
• ✅ More tests: You added tests but didn't optimize the suite (parallelization, markers)
• ✅ Invalidated Docker layers: A change in requirements.txt invalidates the Docker layer cache
• ✅ More AI checks: You added more prompt regression test cases without adjusting the timeouts

Pattern 4: Failures correlated with the time of day

Runs between 9am-5pm UTC: 95% success
Runs between 1am-6am UTC: 70% success

If your failures happen mostly at certain times, the likely causes are:

• ✅ API rate limits at peak hours: The LLM provider has more traffic at certain hours
• ✅ Competing scheduled workflows: If you have nightly runs that run at the same time as other processes
• ✅ GitHub Actions' infrastructure: Shared runners can have performance variations

The GitHub API for pipeline metrics

The visual dashboard is useful for manual inspection. For automated metrics, use the GitHub API.

Listing workflow runs with the API

gh api repos/{owner}/{repo}/actions/workflows/ci.yml/runs \
  --jq '.workflow_runs[:10] | .[] | "\(.id) \(.conclusion) \(.run_started_at) \(.updated_at)"'

The output:

12345678 success 2026-03-07T14:00:00Z 2026-03-07T14:04:32Z
12345677 failure 2026-03-07T12:30:00Z 2026-03-07T12:35:15Z
12345676 success 2026-03-07T10:00:00Z 2026-03-07T10:03:45Z
12345675 success 2026-03-06T16:00:00Z 2026-03-06T16:05:10Z
12345674 failure 2026-03-06T14:30:00Z 2026-03-06T14:38:22Z

The pipeline metrics script

# scripts/pipeline_metrics.py
"""Generate pipeline health metrics using the GitHub API."""

import json
import subprocess
import sys
from datetime import datetime, timezone


def get_workflow_runs(
    owner: str,
    repo: str,
    workflow_file: str = "ci.yml",
    count: int = 50,
) -> list[dict]:
    cmd = [
        "gh", "api",
        f"repos/{owner}/{repo}/actions/workflows/{workflow_file}/runs",
        "--jq", f".workflow_runs[:{count}]",
    ]

    result = subprocess.run(cmd, capture_output=True, text=True)

    if result.returncode != 0:
        print(f"Error: {result.stderr}")
        sys.exit(1)

    return json.loads(result.stdout)


def calculate_metrics(runs: list[dict]) -> dict:
    if not runs:
        return {"error": "No runs found"}

    total = len(runs)
    successes = sum(1 for r in runs if r["conclusion"] == "success")
    failures = sum(1 for r in runs if r["conclusion"] == "failure")
    cancelled = sum(1 for r in runs if r["conclusion"] == "cancelled")

    success_rate = successes / total if total > 0 else 0

    durations = []
    for run in runs:
        if run.get("run_started_at") and run.get("updated_at"):
            start = datetime.fromisoformat(run["run_started_at"].replace("Z", "+00:00"))
            end = datetime.fromisoformat(run["updated_at"].replace("Z", "+00:00"))
            duration_seconds = (end - start).total_seconds()
            durations.append(duration_seconds)

    avg_duration = sum(durations) / len(durations) if durations else 0
    max_duration = max(durations) if durations else 0
    min_duration = min(durations) if durations else 0

    consecutive_failures = 0
    for run in runs:
        if run["conclusion"] == "failure":
            consecutive_failures += 1
        else:
            break

    return {
        "total_runs": total,
        "successes": successes,
        "failures": failures,
        "cancelled": cancelled,
        "success_rate": round(success_rate, 2),
        "avg_duration_seconds": round(avg_duration, 1),
        "max_duration_seconds": round(max_duration, 1),
        "min_duration_seconds": round(min_duration, 1),
        "current_consecutive_failures": consecutive_failures,
        "health": classify_health(success_rate, consecutive_failures),
    }


def classify_health(
    success_rate: float, consecutive_failures: int
) -> str:
    if consecutive_failures >= 3:
        return "CRITICAL"
    if success_rate < 0.7:
        return "UNHEALTHY"
    if success_rate < 0.9:
        return "DEGRADED"
    return "HEALTHY"


def print_report(metrics: dict) -> None:
    health_emoji = {
        "HEALTHY": "✅",
        "DEGRADED": "⚠️",
        "UNHEALTHY": "🔴",
        "CRITICAL": "🚨",
    }

    emoji = health_emoji.get(metrics["health"], "❓")
    print(f"\n{'='*50}")
    print(f"Pipeline Health Report")
    print(f"{'='*50}")
    print(f"")
    print(f"  Status:              {emoji} {metrics['health']}")
    print(f"  Total runs:          {metrics['total_runs']}")
    print(f"  Success rate:        {metrics['success_rate']*100:.0f}%")
    print(f"  Failures:            {metrics['failures']}")
    print(f"  Consecutive fails:   {metrics['current_consecutive_failures']}")
    print(f"")
    print(f"  Avg duration:        {metrics['avg_duration_seconds']:.0f}s")
    print(f"  Min duration:        {metrics['min_duration_seconds']:.0f}s")
    print(f"  Max duration:        {metrics['max_duration_seconds']:.0f}s")
    print(f"{'='*50}")


if __name__ == "__main__":
    import argparse

    parser = argparse.ArgumentParser(description="Pipeline health metrics")
    parser.add_argument("--owner", required=True, help="GitHub org/user")
    parser.add_argument("--repo", required=True, help="Repository name")
    parser.add_argument("--workflow", default="ci.yml", help="Workflow file name")
    parser.add_argument("--count", type=int, default=50, help="Number of runs to analyze")
    parser.add_argument("--json", action="store_true", help="Output as JSON")
    args = parser.parse_args()

    runs = get_workflow_runs(args.owner, args.repo, args.workflow, args.count)
    metrics = calculate_metrics(runs)

    if args.json:
        print(json.dumps(metrics, indent=2))
    else:
        print_report(metrics)

Running it locally

gh auth status

python scripts/pipeline_metrics.py \
  --owner your-username \
  --repo my-ai-project \
  --workflow ci.yml \
  --count 30

The expected output:

==================================================
Pipeline Health Report
==================================================

  Status:              ✅ HEALTHY
  Total runs:          30
  Success rate:        93%
  Failures:            2
  Consecutive fails:   0

  Avg duration:        245s
  Min duration:        180s
  Max duration:        480s
==================================================

Integrating the metrics into the pipeline

You can add the metrics script as a step in your pipeline so it generates a report automatically:

# .github/workflows/pipeline-health.yml
name: Pipeline Health Report

on:
  schedule:
    - cron: "0 9 * * 1"
  workflow_dispatch:

jobs:
  health-report:
    runs-on: ubuntu-latest
    timeout-minutes: 5

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Generate health report
        run: |
          python scripts/pipeline_metrics.py \
            --owner ${{ github.repository_owner }} \
            --repo ${{ github.event.repository.name }} \
            --workflow ci.yml \
            --count 50 \
            --json > pipeline-health.json

          python scripts/pipeline_metrics.py \
            --owner ${{ github.repository_owner }} \
            --repo ${{ github.event.repository.name }} \
            --workflow ci.yml \
            --count 50
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Upload health report
        uses: actions/upload-artifact@v4
        with:
          name: pipeline-health-report
          path: pipeline-health.json
          retention-days: 90

Why every Monday?

A weekly report gives you enough context without generating noise:

• ✅ You see the previous week's trend
• ✅ You detect gradual degradation (growing duration, decreasing success rate)
• ✅ You can act at the start of the week with fresh data

For critical pipelines, you can run the report daily. For most projects, weekly is enough.

Key metrics for AI pipelines

The metrics of AI pipelines have particularities that traditional pipelines don't:

Success rate per check type

Prompt regression has a lower threshold because it depends on external APIs and the LLM's non-determinism. A 90% success rate in prompt regression is healthy — 70% indicates a serious problem.

Duration per stage

A typical stage        A healthy duration    Alert if it exceeds
─────────────────────────────────────────────────────────────
Lint                  15-30s                2 min
Unit tests            30s-2min              5 min
Prompt regression     1-3 min               10 min
Cost estimation       10-30s                2 min
Docker build (cache)  1-3 min               10 min
Docker build (fresh)  3-8 min               15 min
Deploy staging        30s-2 min             5 min
Deploy production     30s-2 min             5 min

If the Docker build regularly exceeds 10 minutes, you probably lost the layer cache. If prompt regression exceeds 10 minutes, you probably have too many test cases or rate limiting.

The cost per run

For AI pipelines, the cost of each run matters because the evaluations use the LLM's API:

Component                Typical cost per run
──────────────────────────────────────────────
The GitHub Actions runner ~$0.04 (8 min × $0.005/min)
LLM calls (10 test cases) ~$0.02
LLM judge (10 evals)      ~$0.12
Docker layer cache         $0 (the GHCR free tier)
──────────────────────────────────────────────
Total without the judge:   ~$0.06 per run
Total with the judge:      ~$0.18 per run

If you do 20 PRs per week: ~$1.20/week without the judge, ~$3.60/week with it. Reasonable, but worth monitoring.

Comparison: Manual vs automated monitoring

Both are complementary: the dashboard for real-time debugging, the script for long-term trends.

Troubleshooting

"The script fails with 'gh: command not found'"

The script uses the gh CLI to access the GitHub API. Install it:

# macOS
brew install gh

# Ubuntu/Debian
sudo apt install gh

# Authenticate
gh auth login

"I don't have enough runs to compute the metrics"

If your repo is new and has fewer than 10 runs, the metrics aren't statistically significant. Wait until you have at least 20-30 runs for the patterns to be visible. In the meantime, monitor manually with the dashboard.

"My runs' duration varies a lot"

GitHub-hosted runners share infrastructure. Variations of ±30% in duration are normal. If the variation is larger, check whether you're using caching — without a cache, every run downloads fresh dependencies and the duration depends on the available bandwidth.

"I want more detailed metrics per job"

The GitHub API lets you query each run's jobs individually:

gh api repos/{owner}/{repo}/actions/runs/{run_id}/jobs \
  --jq '.jobs[] | "\(.name) \(.conclusion) \(.started_at) \(.completed_at)"'

This lets you build per-job metrics: "The ai-checks job has an 85% success rate while lint has 99%." That tells you exactly where to focus your improvement effort.

Exercises

Exercise 1: Analyze your run history

Using your repository's GitHub Actions dashboard, answer:

1. What's your success rate over the last 20 runs?
2. Is there any job that fails more than the others?
3. Has your pipeline gotten slower or faster in the last week?

The last 20 runs: 17 success, 3 failure
Success rate: 85%

Failures:
  Run #142: ai-checks failed (prompt regression)
  Run #138: ai-checks failed (prompt regression)
  Run #135: test failed (timeout)

The pattern: ai-checks is the problematic job (2/3 failures)
The action: Review the prompt regression baselines, a possible model update

Exercise 2: Run the metrics script

Configure and run the pipeline_metrics.py script in your repository:

1. Save the script in scripts/pipeline_metrics.py
2. Run it with --count 20
3. Interpret the result

mkdir -p scripts

python scripts/pipeline_metrics.py \
  --owner your-username \
  --repo my-ai-project \
  --count 20

==================================================
Pipeline Health Report
==================================================

  Status:              ✅ HEALTHY
  Total runs:          20
  Success rate:        90%
  Failures:            2
  Consecutive fails:   0

  Avg duration:        198s
  Min duration:        145s
  Max duration:        312s
==================================================

Exercise 3: Create a health check workflow

Create a workflow that runs the metrics script every Monday and uploads the report as an artifact.

# .github/workflows/pipeline-health.yml
name: Weekly Pipeline Health

on:
  schedule:
    - cron: "0 9 * * 1"
  workflow_dispatch:

jobs:
  health-check:
    runs-on: ubuntu-latest
    timeout-minutes: 5

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Generate report
        run: |
          python scripts/pipeline_metrics.py \
            --owner ${{ github.repository_owner }} \
            --repo ${{ github.event.repository.name }} \
            --count 50 \
            --json > health-report.json

          echo "## Pipeline Health Report" >> $GITHUB_STEP_SUMMARY
          echo "" >> $GITHUB_STEP_SUMMARY

          python -c "
          import json
          with open('health-report.json') as f:
              m = json.load(f)
          print(f'- **Status:** {m[\"health\"]}')
          print(f'- **Success rate:** {m[\"success_rate\"]*100:.0f}%')
          print(f'- **Failures:** {m[\"failures\"]}')
          print(f'- **Avg duration:** {m[\"avg_duration_seconds\"]:.0f}s')
          " >> $GITHUB_STEP_SUMMARY
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Upload report
        uses: actions/upload-artifact@v4
        with:
          name: health-report-${{ github.run_number }}
          path: health-report.json
          retention-days: 90

Exercise 4: Detect duration degradation

Modify the metrics script to compare the average duration of the last 10 runs vs the 10 before them. If the average duration increased by more than 50%, generate an alert.

def detect_duration_degradation(
    runs: list[dict], recent_count: int = 10, threshold: float = 0.5
) -> dict:
    """Compare the duration of recent runs vs earlier ones."""
    durations = []
    for run in runs:
        if run.get("run_started_at") and run.get("updated_at"):
            start = datetime.fromisoformat(run["run_started_at"].replace("Z", "+00:00"))
            end = datetime.fromisoformat(run["updated_at"].replace("Z", "+00:00"))
            durations.append((end - start).total_seconds())

    if len(durations) < recent_count * 2:
        return {
            "status": "INSUFFICIENT_DATA",
            "message": f"Need at least {recent_count * 2} runs, have {len(durations)}",
        }

    recent_avg = sum(durations[:recent_count]) / recent_count
    previous_avg = sum(durations[recent_count:recent_count*2]) / recent_count

    increase_pct = (recent_avg - previous_avg) / previous_avg if previous_avg > 0 else 0

    degraded = increase_pct > threshold

    return {
        "status": "DEGRADED" if degraded else "STABLE",
        "recent_avg_seconds": round(recent_avg, 1),
        "previous_avg_seconds": round(previous_avg, 1),
        "increase_percentage": round(increase_pct * 100, 1),
        "threshold_percentage": threshold * 100,
        "alert": degraded,
    }

{
  "status": "DEGRADED",
  "recent_avg_seconds": 342.5,
  "previous_avg_seconds": 198.3,
  "increase_percentage": 72.7,
  "threshold_percentage": 50.0,
  "alert": true
}

Summary

• ✅ GitHub Actions' dashboard is your main tool for visual monitoring: filter by status, branch, and event
• ✅ Four failure patterns you must recognize: recurring failures in the same job, intermittent (flaky) failures, a gradual degradation in duration, and failures correlated with the time of day
• ✅ The GitHub API lets you extract metrics programmatically: success rate, average duration, consecutive failures
• ✅ The pipeline_metrics.py script automates the pipeline health analysis and classifies it into HEALTHY/DEGRADED/UNHEALTHY/CRITICAL
• ✅ Metrics for AI pipelines have different thresholds: prompt regression has more failure tolerance than lint or unit tests
• ✅ A weekly scheduled workflow generates health reports automatically, detecting trends before they turn into problems
• ✅ Manual and automated monitoring are complementary: the dashboard for specific debugging, scripts for long-term trends

Additional resources

1. GitHub Actions Usage Metrics - The official run history
2. GitHub REST API — Workflow Runs - The API for querying runs
3. GitHub Actions Job Summaries - How to add summaries to runs
4. DORA Metrics - The 4 DevOps metrics: deployment frequency, lead time, failure rate, recovery time
5. GitHub CLI Documentation - GitHub's CLI documentation
6. Flaky Tests at Scale - How Google handles flaky tests