8. Project — AI Quality Gate

Overview

This project integrates everything you learned in Module 3. You're going to build a complete CI pipeline with AI-specific quality gates: lint, type checking, unit tests, prompt regression testing, cost estimation, and a Docker build test — all as required status checks that block the merge if any of them fails.

The result is a GitHub Actions workflow that protects your main branch with checks that no generic CI/CD course teaches.

What you're going to build

PR opened → Workflow triggers → 6 checks run in parallel
                                    │
                                    ├── ✅ Lint (ruff)
                                    ├── ✅ Type Check (mypy)
                                    ├── ✅ Unit Tests (pytest)
                                    ├── ✅ Prompt Regression (keyword eval)
                                    ├── ✅ Cost Estimation (token calc)
                                    └── ✅ Docker Build Test
                                    │
                                    ↓
                              All pass → Merge enabled
                              Any fails → Merge blocked

Artifacts generated:

• Test results (JUnit XML)
• Evaluation logs (JSON)
• Cost report (JSON)

Branch Protection:

• Every check is required
• The branch must be up to date with main
• Admin bypass enabled for emergencies

Project structure

my-ai-project/
├── .github/
│   └── workflows/
│       └── ai-quality-gate.yml
├── src/
│   ├── __init__.py
│   ├── main.py
│   ├── config.py
│   ├── prompts.py
│   └── cost.py
├── tests/
│   ├── __init__.py
│   ├── test_main.py
│   ├── test_prompts.py
│   └── test_cost.py
├── scripts/
│   ├── evaluate_prompts.py
│   └── estimate_cost.py
├── baselines/
│   └── prompt-baseline.json
├── Dockerfile
├── requirements.txt
├── requirements-dev.txt
└── pyproject.toml

Step 1: The base code

src/__init__.py

# src/__init__.py

src/config.py

from dataclasses import dataclass


@dataclass
class Settings:
    app_name: str = "AI Quality Gate Demo"
    model: str = "gpt-4o-mini"
    max_tokens: int = 500
    temperature: float = 0.7
    daily_request_estimate: int = 10000
    cost_threshold_daily: float = 5.00


settings = Settings()

src/prompts.py

SYSTEM_PROMPT = (
    "You are a helpful assistant that provides concise, accurate answers. "
    "Keep responses under 200 words. Use clear language. "
    "If you don't know something, say so honestly."
)

SUMMARY_PROMPT = (
    "Summarize the following text in 2-3 sentences. "
    "Focus on the key points. Be concise and clear."
)

CLASSIFICATION_PROMPT = (
    "Classify the following text into one of these categories: "
    "technical, business, personal, general. "
    "Respond with only the category name."
)


def format_messages(
    user_message: str, system_prompt: str = SYSTEM_PROMPT,
) -> list[dict[str, str]]:
    return [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_message},
    ]


def get_prompt_by_name(name: str) -> str:
    prompts = {"system": SYSTEM_PROMPT, "summary": SUMMARY_PROMPT, "classification": CLASSIFICATION_PROMPT}
    if name not in prompts:
        raise ValueError(f"Unknown prompt: {name}. Available: {list(prompts.keys())}")
    return prompts[name]


def count_prompt_tokens(prompt: str, model: str = "gpt-4o-mini") -> int:
    return int(len(prompt.split()) * 1.3)

PRICING = {
    "gpt-4o-mini": {"input_per_1m": 0.15, "output_per_1m": 0.60},
    "gpt-4o": {"input_per_1m": 2.50, "output_per_1m": 10.00},
    "gpt-4.1-mini": {"input_per_1m": 0.40, "output_per_1m": 1.60},
}


def estimate_request_cost(
    input_tokens: int, output_tokens: int, model: str = "gpt-4o-mini",
) -> float:
    if model not in PRICING:
        raise ValueError(f"Unknown model: {model}. Available: {list(PRICING.keys())}")
    prices = PRICING[model]
    input_cost = (input_tokens / 1_000_000) * prices["input_per_1m"]
    output_cost = (output_tokens / 1_000_000) * prices["output_per_1m"]
    return round(input_cost + output_cost, 8)


def estimate_daily_cost(
    input_tokens: int, output_tokens: int,
    requests_per_day: int, model: str = "gpt-4o-mini",
) -> float:
    cost_per_request = estimate_request_cost(input_tokens, output_tokens, model)
    return round(cost_per_request * requests_per_day, 2)


def check_cost_threshold(daily_cost: float, threshold: float) -> dict:
    status = "PASS" if daily_cost <= threshold else "FAIL"
    return {
        "daily_cost": daily_cost, "threshold": threshold,
        "status": status, "delta": round(daily_cost - threshold, 2),
    }

from fastapi import FastAPI
from pydantic import BaseModel
from src.config import settings
from src.prompts import get_prompt_by_name
from src.cost import estimate_request_cost

app = FastAPI(title=settings.app_name)


class HealthResponse(BaseModel):
    status: str
    app_name: str


class CostRequest(BaseModel):
    input_tokens: int
    output_tokens: int
    model: str = "gpt-4o-mini"


class CostResponse(BaseModel):
    cost: float
    model: str


@app.get("/health", response_model=HealthResponse)
def health_check() -> HealthResponse:
    return HealthResponse(status="healthy", app_name=settings.app_name)


@app.post("/estimate-cost", response_model=CostResponse)
def estimate_cost(request: CostRequest) -> CostResponse:
    cost = estimate_request_cost(
        input_tokens=request.input_tokens,
        output_tokens=request.output_tokens,
        model=request.model,
    )
    return CostResponse(cost=cost, model=request.model)


@app.get("/prompts/{name}")
def get_prompt(name: str) -> dict:
    return {"name": name, "prompt": get_prompt_by_name(name)}

Step 2: The tests

tests/__init__.py

# tests/__init__.py

tests/test_main.py

from fastapi.testclient import TestClient
from src.main import app

client = TestClient(app)


def test_health_check():
    response = client.get("/health")
    assert response.status_code == 200
    data = response.json()
    assert data["status"] == "healthy"
    assert "app_name" in data


def test_estimate_cost_endpoint():
    response = client.post(
        "/estimate-cost",
        json={"input_tokens": 100, "output_tokens": 200, "model": "gpt-4o-mini"},
    )
    assert response.status_code == 200
    data = response.json()
    assert "cost" in data
    assert data["cost"] > 0
    assert data["model"] == "gpt-4o-mini"


def test_estimate_cost_invalid_model():
    response = client.post(
        "/estimate-cost",
        json={"input_tokens": 100, "output_tokens": 200, "model": "invalid-model"},
    )
    assert response.status_code == 500


def test_get_prompt():
    response = client.get("/prompts/system")
    assert response.status_code == 200
    data = response.json()
    assert data["name"] == "system"
    assert len(data["prompt"]) > 0


def test_get_prompt_not_found():
    response = client.get("/prompts/nonexistent")
    assert response.status_code == 500

import pytest
from src.prompts import (
    SYSTEM_PROMPT,
    SUMMARY_PROMPT,
    CLASSIFICATION_PROMPT,
    format_messages,
    get_prompt_by_name,
    count_prompt_tokens,
)


def test_system_prompt_exists():
    assert len(SYSTEM_PROMPT) > 0
    assert "helpful" in SYSTEM_PROMPT.lower()


def test_summary_prompt_exists():
    assert len(SUMMARY_PROMPT) > 0
    assert "summarize" in SUMMARY_PROMPT.lower()


def test_classification_prompt_exists():
    assert len(CLASSIFICATION_PROMPT) > 0
    assert "classify" in CLASSIFICATION_PROMPT.lower()


def test_format_messages_default():
    messages = format_messages("Hello")
    assert len(messages) == 2
    assert messages[0]["role"] == "system"
    assert messages[0]["content"] == SYSTEM_PROMPT
    assert messages[1]["role"] == "user"
    assert messages[1]["content"] == "Hello"


def test_format_messages_custom_system():
    custom = "You are a pirate."
    messages = format_messages("Hello", system_prompt=custom)
    assert messages[0]["content"] == custom


def test_get_prompt_by_name_valid():
    assert get_prompt_by_name("system") == SYSTEM_PROMPT
    assert get_prompt_by_name("summary") == SUMMARY_PROMPT
    assert get_prompt_by_name("classification") == CLASSIFICATION_PROMPT


def test_get_prompt_by_name_invalid():
    with pytest.raises(ValueError, match="Unknown prompt"):
        get_prompt_by_name("nonexistent")


def test_count_prompt_tokens():
    prompt = "This is a test prompt with several words"
    tokens = count_prompt_tokens(prompt)
    assert tokens > 0
    assert isinstance(tokens, int)


def test_count_prompt_tokens_empty():
    tokens = count_prompt_tokens("")
    assert tokens == 0

import pytest
from src.cost import (
    estimate_request_cost,
    estimate_daily_cost,
    check_cost_threshold,
    PRICING,
)


def test_estimate_request_cost_basic():
    cost = estimate_request_cost(1000, 500, "gpt-4o-mini")
    assert cost > 0
    assert isinstance(cost, float)


def test_estimate_request_cost_gpt4o():
    cost_mini = estimate_request_cost(1000, 500, "gpt-4o-mini")
    cost_4o = estimate_request_cost(1000, 500, "gpt-4o")
    assert cost_4o > cost_mini


def test_estimate_request_cost_zero_tokens():
    cost = estimate_request_cost(0, 0, "gpt-4o-mini")
    assert cost == 0.0


def test_estimate_request_cost_invalid_model():
    with pytest.raises(ValueError, match="Unknown model"):
        estimate_request_cost(1000, 500, "invalid-model")


def test_estimate_daily_cost():
    daily = estimate_daily_cost(100, 200, 10000, "gpt-4o-mini")
    assert daily > 0
    assert isinstance(daily, float)


def test_estimate_daily_cost_zero_requests():
    daily = estimate_daily_cost(100, 200, 0, "gpt-4o-mini")
    assert daily == 0.0


def test_check_cost_threshold_pass():
    result = check_cost_threshold(3.00, 5.00)
    assert result["status"] == "PASS"
    assert result["delta"] < 0


def test_check_cost_threshold_fail():
    result = check_cost_threshold(7.00, 5.00)
    assert result["status"] == "FAIL"
    assert result["delta"] > 0


def test_check_cost_threshold_exact():
    result = check_cost_threshold(5.00, 5.00)
    assert result["status"] == "PASS"
    assert result["delta"] == 0


def test_pricing_models_exist():
    assert "gpt-4o-mini" in PRICING
    assert "gpt-4o" in PRICING
    for model, prices in PRICING.items():
        assert "input_per_1m" in prices
        assert "output_per_1m" in prices

Step 3: The evaluation scripts

scripts/evaluate_prompts.py

#!/usr/bin/env python3
"""Prompt regression testing — deterministic evaluation without API keys."""
import json, sys
from pathlib import Path
from datetime import datetime, timezone

project_root = Path(__file__).parent.parent
sys.path.insert(0, str(project_root))
from src.prompts import get_prompt_by_name, count_prompt_tokens


def evaluate_prompt(prompt_name: str, prompt_text: str, expected: dict) -> dict:
    result = {"prompt_name": prompt_name, "passed": True, "checks": []}

    for keyword in expected.get("required_keywords", []):
        found = keyword.lower() in prompt_text.lower()
        result["checks"].append({"type": "keyword_present", "keyword": keyword, "found": found})
        if not found:
            result["passed"] = False

    for keyword in expected.get("forbidden_keywords", []):
        found = keyword.lower() in prompt_text.lower()
        result["checks"].append({"type": "keyword_absent", "keyword": keyword, "found": found})
        if found:
            result["passed"] = False

    if "max_tokens" in expected:
        token_count = count_prompt_tokens(prompt_text)
        within_limit = token_count <= expected["max_tokens"]
        result["checks"].append({"type": "max_tokens", "limit": expected["max_tokens"],
            "actual": token_count, "within_limit": within_limit})
        if not within_limit:
            result["passed"] = False

    if "min_length" in expected:
        meets_min = len(prompt_text) >= expected["min_length"]
        result["checks"].append({"type": "min_length", "minimum": expected["min_length"],
            "actual": len(prompt_text), "meets_minimum": meets_min})
        if not meets_min:
            result["passed"] = False

    return result


def run_evaluation(baseline_path: str, output_path: str) -> bool:
    with open(baseline_path) as f:
        baseline = json.load(f)

    results = []
    all_passed = True

    for test_case in baseline["test_cases"]:
        prompt_name = test_case["prompt_name"]
        try:
            prompt_text = get_prompt_by_name(prompt_name)
        except ValueError as e:
            results.append({"prompt_name": prompt_name, "passed": False, "error": str(e), "checks": []})
            all_passed = False
            continue
        result = evaluate_prompt(prompt_name, prompt_text, test_case["expected"])
        results.append(result)
        if not result["passed"]:
            all_passed = False

    total = len(results)
    passed = sum(1 for r in results if r["passed"])
    output = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "baseline_file": baseline_path, "evaluations": results,
        "summary": {"total": total, "passed": passed, "failed": total - passed,
                     "pass_rate": passed / total if total > 0 else 0},
    }

    output_file = Path(output_path)
    output_file.parent.mkdir(parents=True, exist_ok=True)
    with open(output_file, "w") as f:
        json.dump(output, f, indent=2)

    print(f"Evaluation complete: {passed}/{total} passed")
    return all_passed


if __name__ == "__main__":
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument("--baseline", default="baselines/prompt-baseline.json")
    parser.add_argument("--output", default="results/evaluation.json")
    args = parser.parse_args()
    sys.exit(0 if run_evaluation(args.baseline, args.output) else 1)

#!/usr/bin/env python3
"""Cost estimation check for CI. It requires no API keys."""
import json
import sys
from pathlib import Path
from datetime import datetime, timezone

project_root = Path(__file__).parent.parent
sys.path.insert(0, str(project_root))

from src.prompts import get_prompt_by_name, count_prompt_tokens
from src.cost import estimate_request_cost, estimate_daily_cost, check_cost_threshold
from src.config import settings


def estimate_all_prompts(
    model: str,
    estimated_output_tokens: int = 200,
    requests_per_day: int = 10000,
) -> list[dict]:
    prompt_names = ["system", "summary", "classification"]
    estimates = []

    for name in prompt_names:
        prompt_text = get_prompt_by_name(name)
        input_tokens = count_prompt_tokens(prompt_text)
        cost_per_request = estimate_request_cost(input_tokens, estimated_output_tokens, model)
        daily_cost = estimate_daily_cost(input_tokens, estimated_output_tokens, requests_per_day, model)

        estimates.append({
            "prompt_name": name,
            "input_tokens": input_tokens,
            "estimated_output_tokens": estimated_output_tokens,
            "model": model,
            "cost_per_request": cost_per_request,
            "daily_cost": daily_cost,
        })

    return estimates


def run_cost_estimation(output_path: str, threshold: float) -> bool:
    estimates = estimate_all_prompts(
        model=settings.model,
        requests_per_day=settings.daily_request_estimate,
    )

    total_daily = sum(e["daily_cost"] for e in estimates)
    threshold_check = check_cost_threshold(total_daily, threshold)

    report = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "model": settings.model,
        "requests_per_day": settings.daily_request_estimate,
        "prompts": estimates,
        "total_daily_cost": total_daily,
        "threshold": threshold,
        "status": threshold_check["status"],
        "delta": threshold_check["delta"],
    }

    output_file = Path(output_path)
    output_file.parent.mkdir(parents=True, exist_ok=True)
    with open(output_file, "w") as f:
        json.dump(report, f, indent=2)

    print(f"Cost estimation: ${total_daily:.2f}/day | Threshold: ${threshold:.2f} | {threshold_check['status']}")
    return threshold_check["status"] == "PASS"


if __name__ == "__main__":
    import argparse
    parser = argparse.ArgumentParser(description="Cost estimation check")
    parser.add_argument("--output", default="results/cost-report.json")
    parser.add_argument("--threshold", type=float, default=settings.cost_threshold_daily)
    args = parser.parse_args()
    success = run_cost_estimation(args.output, args.threshold)
    sys.exit(0 if success else 1)

{
  "version": "1.0",
  "description": "Baseline for prompt regression testing",
  "test_cases": [
    {
      "prompt_name": "system",
      "expected": {
        "required_keywords": ["helpful", "concise", "accurate"],
        "forbidden_keywords": ["ignore previous instructions", "jailbreak"],
        "max_tokens": 100, "min_length": 50
      }
    },
    {
      "prompt_name": "summary",
      "expected": {
        "required_keywords": ["summarize", "concise"],
        "forbidden_keywords": ["ignore", "override"],
        "max_tokens": 80, "min_length": 30
      }
    },
    {
      "prompt_name": "classification",
      "expected": {
        "required_keywords": ["classify", "category"],
        "forbidden_keywords": ["ignore previous"],
        "max_tokens": 80, "min_length": 30
      }
    }
  ]
}

Step 4: Project configuration

requirements.txt

fastapi>=0.110.0
uvicorn>=0.29.0
pydantic>=2.6.0

requirements-dev.txt

-r requirements.txt
pytest>=8.0
pytest-timeout>=2.3
pytest-cov>=5.0
httpx>=0.27.0
ruff>=0.4.0
mypy>=1.9.0

[project]
name = "ai-quality-gate-demo"
version = "0.1.0"
requires-python = ">=3.10"

[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-v --tb=short"
timeout = 30
timeout_method = "signal"

[tool.ruff]
target-version = "py312"
line-length = 100
select = ["E", "F", "I", "N", "W", "UP"]

[tool.ruff.lint.isort]
known-first-party = ["src"]

[tool.mypy]
python_version = "3.12"
warn_return_any = true
warn_unused_configs = true
ignore_missing_imports = true

[tool.coverage.run]
source = ["src"]

[tool.coverage.report]
show_missing = true
fail_under = 80

Dockerfile

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY src/ ./src/
EXPOSE 8000

HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
  CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')" || exit 1

CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"]

Step 5: The GitHub Actions workflow

.github/workflows/ai-quality-gate.yml

name: AI Quality Gate

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  lint:
    name: Lint & Type Check
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements-dev.txt
      - run: ruff check src/ tests/ scripts/
      - run: mypy src/ --ignore-missing-imports

  test:
    name: Unit Tests
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements-dev.txt
      - name: Run unit tests with coverage
        run: |
          mkdir -p results/tests
          pytest tests/ -v --timeout=30 \
            --junitxml=results/tests/report.xml \
            --cov=src --cov-report=term-missing --cov-fail-under=80
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: test-results
          path: results/tests/
          retention-days: 30

  prompt-regression:
    name: Prompt Regression
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - run: |
          python scripts/evaluate_prompts.py \
            --baseline baselines/prompt-baseline.json \
            --output results/evaluation.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: evaluation-results
          path: results/evaluation.json
          retention-days: 30

  cost-estimation:
    name: Cost Estimation
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - run: |
          python scripts/estimate_cost.py \
            --output results/cost-report.json \
            --threshold 5.00
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: cost-report
          path: results/cost-report.json
          retention-days: 14

  docker-build:
    name: Docker Build
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t ai-quality-gate:${{ github.sha }} .
      - run: |
          docker run --rm ai-quality-gate:${{ github.sha }} python -c "
          from src.main import app
          from src.config import settings
          from src.prompts import SYSTEM_PROMPT
          from src.cost import PRICING
          print('All imports OK')
          print(f'App: {settings.app_name}')
          "

  summary:
    name: Pipeline Summary
    runs-on: ubuntu-latest
    needs: [lint, test, prompt-regression, cost-estimation, docker-build]
    if: always()
    steps:
      - uses: actions/download-artifact@v4
        with:
          path: all-results/
        continue-on-error: true
      - name: Generate pipeline summary
        run: |
          echo "## AI Quality Gate Results" >> $GITHUB_STEP_SUMMARY
          echo "| Check | Status |" >> $GITHUB_STEP_SUMMARY
          echo "|-------|--------|" >> $GITHUB_STEP_SUMMARY
          echo "| Lint | ${{ needs.lint.result }} |" >> $GITHUB_STEP_SUMMARY
          echo "| Tests | ${{ needs.test.result }} |" >> $GITHUB_STEP_SUMMARY
          echo "| Prompts | ${{ needs.prompt-regression.result }} |" >> $GITHUB_STEP_SUMMARY
          echo "| Cost | ${{ needs.cost-estimation.result }} |" >> $GITHUB_STEP_SUMMARY
          echo "| Docker | ${{ needs.docker-build.result }} |" >> $GITHUB_STEP_SUMMARY

Step 6: Configure Branch Protection

After pushing the workflow and letting it run at least once on main:

Settings → Branches → Add branch protection rule
  Branch name pattern: main
  ☑ Require a pull request before merging (1 approval)
  ☑ Require status checks to pass before merging
    Required: Lint & Type Check, Unit Tests, Prompt Regression, Cost Estimation, Docker Build
  ☑ Require conversation resolution before merging
  ☐ Do not allow bypassing (leave bypass for admins)
  → Create

The Pipeline Summary job is not required — it's informational.

Step 7: Test the quality gate

Create a branch for each test, open a PR, and verify that the merge is blocked:

git checkout -b test/break-lint
# Make the corresponding change
git add . && git commit -m "Test: trigger failure" && git push origin HEAD
# Open a PR and verify that the merge is blocked

Step 8: Verify the artifacts

After the workflow runs, go to Actions → workflow run → the Artifacts section:

• test-results — JUnit XML
• evaluation-results — JSON with the prompt results
• cost-report — JSON with the cost report

gh run list --limit 5
gh run download $(gh run list --limit 1 --json databaseId -q '.[0].databaseId')

Troubleshooting

1. The scripts can't find the modules in src/

Cause: Python can't resolve the import because src/ isn't on the path.

Solution: The scripts include sys.path.insert(0, str(project_root)). In CI, actions/checkout@v4 places the runner at the repo root automatically.

2. mypy reports errors in third-party dependencies

Solution: --ignore-missing-imports is already in the workflow. For more control:

[[tool.mypy.overrides]]
module = ["fastapi.*", "uvicorn.*"]
ignore_missing_imports = true

3. The cost estimation passes locally but fails in CI

Cause: Tiny differences in the token estimate (rounding).

Solution: Use a threshold with some margin (--threshold 5.50) or adjust the script so it doesn't fail over differences of a few cents.

Success criteria

Your project is complete when:

1. The workflow has 6 checks that run in parallel
2. The 5 main checks are required in Branch Protection (summary is informational)
3. Breaking the lint blocks the merge
4. Modifying a prompt (removing an expected keyword) blocks the merge
5. Switching to an expensive model blocks the merge
6. The artifacts get generated and are downloadable
7. The Pipeline Summary shows a readable summary
8. All the tests pass locally with pytest tests/ -v
9. The Docker build works with docker build -t test .

Connection with Module 4

Your pipeline works for deterministic checks — lint, type check, keyword matching, token counting, Docker build. None of them needs an API key. Module 4 — Secrets and Environment Management — solves how to handle API keys in CI for prompt regression with a real LLM, integration tests, and per-environment configurations.

Summary

• ✅ A complete pipeline with 6 checks: lint, type check, unit tests, prompt regression, cost estimation, Docker build
• ✅ The checks run in parallel — each one is an independent job
• ✅ Deterministic prompt regression testing — keyword matching without API keys
• ✅ Local cost estimation — it counts tokens and cost without calling the API
• ✅ Artifacts generated: test results (JUnit XML), evaluation logs (JSON), cost report (JSON)
• ✅ The Pipeline Summary consolidates the results into a readable Job Summary
• ✅ Branch Protection with 5 required status checks blocks the merge if any check fails
• ✅ Scripts runnable locally — python scripts/evaluate_prompts.py and python scripts/estimate_cost.py

Additional resources

1. GitHub Actions workflow syntax — Complete reference of the YAML syntax
2. Branch protection rules — How to configure branch protection
3. ruff configuration — Linter configuration reference
4. GitHub Actions Job Summaries — How to publish Markdown in the Job Summary