Module 3: AI-Specific CI Checks

1. Introduction: AI-Specific CI Checks

Next →

Overview

Modules 1 and 2 gave you a working CI pipeline: workflows with GitHub Actions, automated pytest, matrix testing, caching, coverage reports, and timeouts. All of that is professional CI. But it's also generic CI — any Python project needs it, whether it's a web app, a CLI, or a microservice.

This module is different. Here you add checks that only exist in the context of AI applications. Prompt regression testing, cost estimation, and linting configuration specific to projects with LLMs. No generic CI/CD course teaches this. It's the reason this guide is called "CI/CD for AI Systems" and not simply "CI/CD with GitHub Actions."

The central problem: In traditional software, a test passes or fails — it's binary. In AI systems, you can change a prompt and every test passes, but the quality of the responses degrades silently. A refactor that modifies the system prompt can produce responses that are technically correct but less useful, more expensive, or inconsistent. Unit tests don't detect that. Prompt regression testing does.

The second problem: A badly designed prompt can multiply your costs by 10x. Without an automated check in CI, nobody finds out until the OpenAI invoice arrives at the end of the month. A quality gate that blocks a merge because the new prompt costs $0.15 per request (vs $0.02 for the previous one) is a financial protection that doesn't exist in traditional CI.

Where Are We in the Guide?

Context

This guide has 8 modules organized into 3 phases:

Phase 1: CI Fundamentals (Modules 1-3)
├── Module 1: Introduction to CI/CD and GitHub Actions    ✅ Completed
├── Module 2: Automated Testing in CI                     ✅ Completed
└── Module 3: AI-Specific CI Checks                       ← YOU ARE HERE

Phase 2: CD & Deployment Pipelines (Modules 4-6)
├── Module 4: Secrets and Environment Management
├── Module 5: Docker in CI/CD
└── Module 6: Deployment Pipelines

Phase 3: Production Pipelines (Modules 7-8)
├── Module 7: Monitoring, Notifications and Advanced Patterns
└── Module 8: Capstone Project — Production AI Pipeline

Estimated duration of the module: 60-90 minutes.

Where are we headed?

This module closes Phase 1 (CI Fundamentals). After here, you move on to Phase 2 where everything you built gets connected to secrets, Docker, and deployment. The progression is deliberate:

  1. First you understood CI (module 1) — workflows, jobs, steps, triggers
  2. Then you automated testing (module 2) — pytest, matrix, caching, reports
  3. Now you add AI-specific checks (this module) — prompt regression, cost estimation, linting
  4. Then you handle secrets (module 4) — secure API keys in CI for this module's checks
  5. You build Docker images (module 5) — automated containerization
  6. You deploy to staging and prod (module 6) — deployment pipelines
  7. You add monitoring and patterns (module 7) — notifications, scheduled workflows
  8. You integrate everything (module 8) — a complete commit-to-production pipeline

Context: Where are we?

ModuleWhat you learned/will learn
Module 1CI/CD concepts, GitHub Actions basics, first workflow
Module 2pytest in CI, matrix testing, caching, reports, timeouts
Module 3AI-specific checks: prompt regression, cost estimation, linting
Module 4Secrets and environment management
Module 5Docker in CI/CD
Module 6Deployment pipelines
Module 7Advanced patterns
Module 8Capstone project

The progression

Modules 1-2:  Generic CI (any Python project)
    ↓
Module 3:     CI specific to AI (this module)
    ↓
Modules 4-8:  CD, deployment, production

Module 3 is the bridge between "I have CI that runs tests" and "I have CI that protects the quality and the cost of my AI system." Everything you built in modules 1-2 keeps working — here you add layers of protection that are exclusive to applications with LLMs.

What makes CI for AI different

Traditional CI vs CI for AI Systems

AspectTraditional CICI for AI Systems
TestsUnit tests, integration tests+ Prompt regression tests
QualityCode coverage, linting+ Quality of the LLM's responses
CostsNot monitored in CICost estimation per prompt
RegressionA test passes or failsA response can degrade gradually
BaselineNot applicableQuality and cost baselines you compare against
Type checkingStandard mypymypy configured for AI libs without type stubs

The pipeline you're going to build

By the end of this module, your CI workflow looks like this:

Push / PR
    ↓
┌─────────────────────────────────┐
│  Job: AI Quality Gate           │
│                                 │
│  1. Checkout + Setup            │
│  2. Install dependencies        │
│  3. Lint (ruff)                 │
│  4. Type check (mypy)           │
│  5. Unit tests (pytest)         │
│  6. Prompt regression testing   │  ← AI-specific
│  7. Cost estimation check       │  ← AI-specific
│                                 │
│  If any fails → ❌ Block        │
└─────────────────────────────────┘

Steps 3-5 are the generic CI you already know. Steps 6-7 are the new part — and they're what makes your pipeline AI-aware.

Goal of the module

By completing this module you will be able to:

  • ✅ Implement prompt regression testing in CI: test cases, automated evaluations, comparison against a baseline
  • ✅ Configure cost estimation checks: count tokens, estimate costs, fail if a prompt becomes too expensive
  • ✅ Integrate ruff and mypy into CI with configuration specific to AI projects
  • ✅ Configure mypy so it works with libraries that have no type stubs (openai, langchain)
  • ✅ Design a workflow that combines generic checks with AI-specific checks
  • ✅ Understand when a check must block a merge and when it's informational

The professional goal

When your team modifies a prompt in a PR, your CI will automatically: (1) verify that the quality of the responses didn't degrade, (2) estimate whether the cost per request changed significantly, and (3) report the results. If something doesn't meet the thresholds, the merge is blocked. That's CI for AI systems in production.

Prerequisites

  • ✅ Module 1 completed: You know how to create workflows, you understand jobs, steps, triggers
  • ✅ Module 2 completed: pytest runs in CI with matrix testing and caching
  • ✅ Intermediate Python: Functions, classes, JSON, f-strings
  • ✅ Basic experience with LLM APIs: You've used the OpenAI API at least once

Module contents

Capsule 02: Prompt Regression Testing in CI

The killer feature of this module. You build a complete prompt regression testing system: you define test cases with inputs and expected baselines, you implement evaluations (keyword matching, semantic similarity, LLM-as-judge), and you integrate it as a step in GitHub Actions that fails if the quality of the responses degrades after a prompt change.

Capsule 03: Cost Estimation Checks

The financial guardrail. You implement a script that calculates the estimated cost of every prompt based on token count and the model's pricing, compares it against the baseline cost of the previous prompt, and blocks the merge if the increase exceeds a threshold. A prompt that goes from $0.02 to $0.15 per request doesn't reach production without someone consciously approving it.

Capsule 04: Linting and Type Checking in CI

The foundation of code quality. You configure ruff and mypy as CI steps, with configuration specific to AI projects: the relevant ruff rules, mypy configured not to complain about libraries without type stubs (openai, langchain, tiktoken), and how to handle the gap between what works locally and what fails in CI.

Connection with the guide's project

This module's project: AI Quality Gate

The mini-project integrates every check into a single workflow:

  1. Lint with ruff — code style and errors
  2. Type check with mypy — correct types
  3. Unit tests with pytest — functional logic
  4. Prompt regression testing — response quality
  5. Cost estimation — cost control

If any check fails, the PR can't be merged. The results of prompt regression and cost estimation are saved as artifacts for debugging.

Connection with later modules

Module 3: AI Quality Gate (checks)
    ↓
Module 4: Secrets management (API keys for the checks)
    ↓
Module 5: Docker build test (another check in the pipeline)
    ↓
Module 8: Capstone pipeline (all the checks + deploy)

Module 4 solves a problem that emerges directly from this module: to run prompt regression tests in CI, you need an OpenAI API key on the runner. How do you handle it safely? That's the natural transition.

What carries over from Module 2

Your current pipeline (after Module 2) looks like this:

# The current state of the pipeline
name: CI Pipeline
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.10", "3.11", "3.12"]
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: "pip"
      - run: pip install -r requirements.txt
      - run: pytest tests/ -v --tb=short --junitxml=report.xml --cov=src
      - uses: actions/upload-artifact@v4
        with:
          name: test-report-${{ matrix.python-version }}
          path: report.xml

Functional. Professional. But blind to prompt quality and to the cost of LLM calls. This module adds those eyes.

What this module adds

# The state of the pipeline AFTER this module
name: AI Quality Gate
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  quality-gate:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements-dev.txt

      # Generic checks (Modules 1-2)
      - name: Lint with ruff
        run: ruff check src/ tests/ scripts/
      - name: Type check with mypy
        run: mypy src/ scripts/
      - name: Run tests
        run: pytest tests/ -v --tb=short

      # AI-specific checks (Module 3 — NEW)
      - name: Cost estimation
        run: python scripts/estimate_costs.py
      - name: Prompt regression testing
        run: python scripts/evaluate_prompts.py
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

The difference between Module 2's pipeline and this module's: two new steps that protect quality and costs. Everything else stays the same — you're only adding layers.

Technical setup

New dependencies for this module

# In your project, install the new dependencies
pip install tiktoken openai python-dotenv ruff mypy
# requirements-dev.txt (add to your existing file)
-r requirements.txt
pytest==8.3.4
pytest-cov==6.0.0
ruff==0.9.2
mypy==1.14.1
tiktoken==0.8.0

The module's file structure

your-ai-project/
├── .github/
│   └── workflows/
│       └── ai-quality-gate.yml    # Workflow with the AI checks
├── config/
│   ├── pricing.json               # Prices per model (cost estimation)
│   └── prompts.json               # The project's registered prompts
├── scripts/
│   ├── evaluate_prompts.py        # Prompt regression script
│   └── estimate_costs.py          # Cost estimation script
├── src/
│   └── ai_app/
│       ├── __init__.py
│       └── chain.py               # Your main code
├── tests/
│   ├── prompt_test_cases.json     # Test cases for prompt regression
│   ├── conftest.py
│   └── test_chain.py
├── pyproject.toml                 # ruff, mypy, pytest config
└── requirements-dev.txt           # Development dependencies

Quick verification

Before starting the capsules, verify that the tools are installed:

# Verify ruff
ruff --version
# Expected output: ruff 0.9.2

# Verify mypy
mypy --version
# Expected output: mypy 1.14.1 (compiled: yes)

# Verify tiktoken
python -c "import tiktoken; print(tiktoken.encoding_for_model('gpt-4o-mini'))"
# Expected output: <Encoding 'o200k_base'>

If all three commands work, you're ready for the module.

The AI-specific checks you're going to implement

Check 1: Prompt Regression Testing

What does it do?
  It runs a set of test cases against your current prompt
  and evaluates whether the quality of the responses is acceptable.

When does it run?
  On every PR that modifies prompts or the app's code.

What does it detect?
  - Responses that no longer contain the expected keywords
  - Responses that are too short or too long
  - Quality degradation vs the previous baseline

How much does it cost?
  ~$0.01-0.12 per run (depends on the number of test cases and the model)

Does it need an API key?
  Yes — it calls the LLM to generate the responses.

Check 2: Cost Estimation

What does it do?
  It calculates how many tokens your current prompt uses
  and estimates the cost per request.

When does it run?
  On every PR that modifies prompt files or the pricing config.

What does it detect?
  - Cost increases larger than the defined threshold
  - Prompts that exceed an absolute maximum cost

How much does it cost?
  $0 — it uses tiktoken locally, with no API calls.

Does it need an API key?
  No — everything is calculated offline with tiktoken.

Check 3: Linting and Type Checking

What does it do?
  It verifies code style (ruff) and types (mypy)
  with configuration specific to AI projects.

When does it run?
  On every push and PR.

What does it detect?
  - Unused imports, bare excepts, mutable defaults
  - Type errors that would cause runtime crashes
  - Code without type hints

How much does it cost?
  $0 — local tools.

Does it need an API key?
  No.

Execution order in CI

Fastest and cheapest ──────────────────────── Slowest and most expensive

  ruff        mypy       pytest      cost est.    prompt regression
  (<1s, $0)   (<5s, $0)  (<30s, $0)  (<2s, $0)   (<60s, ~$0.10)
  
  ───────── If any fails, the following ones do NOT run ─────────

The order is not arbitrary. The free and fast checks go first. If ruff detects a missing import in 0.5 seconds, there's no point spending $0.10 on prompt regression. Each check is a filter that saves time and money when it fails early.

What this module does NOT cover

  • ❌ Secrets management: How to handle API keys in CI safely is Module 4. Here we assume the secret exists.
  • ❌ Docker build in CI: Verifying that the Dockerfile builds is Module 5.
  • ❌ Detailed branch protection rules: We mention the concept, but the complete configuration of GitHub branch protection is part of the capstone project (Module 8).
  • ❌ Evaluations with fine-tuned models: We use standard models (gpt-4o-mini) for evaluations. Fine-tuning evaluators is out of scope.
  • ❌ Monitoring in production: Cost estimation here is pre-merge. Monitoring costs in production is guide #18.

The mindset shift

Before this module

Developer: "I changed the system prompt so it's more concise"
CI: ✅ Tests pass, ✅ Lint clean
Developer: *merge*
A week later: "Why are the chatbot's responses worse?"
A month later: "Why did the OpenAI invoice go up 300%?"

After this module

Developer: "I changed the system prompt so it's more concise"
CI: ✅ Tests pass, ✅ Lint clean
CI: ❌ Prompt regression: quality dropped 23% vs baseline
CI: ⚠️ Cost estimation: +180% tokens per request
Developer: "I'm going to review the prompt before merging"

That's the difference between generic CI and CI for AI systems.

Analogy: Quality control in a factory

Imagine a car factory. Traditional CI is like verifying that the parts fit, that the bolts are tightened, and that the engine starts. That's fine for a conventional car.

But if the factory produces autonomous electric cars, you need extra checks: does the AI navigation system respond correctly to traffic signs? Is the processing cost per decision within budget? Are the sensors calibrated to the standard?

The generic checks (bolts, engine) are still necessary. But without the AI-specific checks (navigation, cost, calibration), the car could leave the factory "working" but making the wrong decisions.

Your AI application is the same. pytest verifies that the functions return what's expected (the bolts). Prompt regression testing verifies that the LLM's responses keep their quality (the navigation). Cost estimation verifies that you're not spending 10x more than necessary (the budget).

Quick self-assessment

Before starting, verify that you have the necessary context. If you can answer these questions, you're ready:

  1. What is a GitHub Actions workflow and what is a job? (Module 1)
  2. How do you configure pytest to run in CI with matrix testing? (Module 2)
  3. What is an artifact in GitHub Actions? (Module 2)
  4. What does actions/upload-artifact@v4 do? (Module 2)
  5. Why would you use continue-on-error: true on a step? (Module 2)

If any question sounds completely new, review modules 1-2 before continuing. This module assumes you have those concepts down.

Evidence of success

By the end of this module, you should be able to:

  • Define test cases for prompt regression with inputs and expected baselines
  • Implement deterministic evaluations (keyword matching) and LLM-as-judge ones
  • Integrate prompt regression testing as a GitHub Actions step
  • Calculate the estimated cost of a prompt using token count and pricing
  • Configure a cost threshold that blocks the merge if it's exceeded
  • Configure ruff and mypy in CI with settings specific to AI projects
  • Design a workflow that combines generic and AI-specific checks

If you tick every check → you're ready for Module 4.

Additional resources

  1. GitHub Actions — Creating Custom Actions — For creating reusable actions with your AI checks
  2. OpenAI Tokenizer — A visual tool for understanding tokenization
  3. tiktoken — OpenAI Token Counter — Library for counting tokens programmatically
  4. ruff — Python Linter — Complete ruff documentation
  5. mypy — Static Type Checker — Configuring mypy for complex projects
  6. OpenAI Pricing — Up-to-date prices per model and per token
← Previous8. Project — Automated Test PipelineNext →2. Prompt Regression Testing in CI