4. Security Deployment Checklist

Overview

An AI system that passes all the tests in staging can be vulnerable in production if the deploy isn't executed correctly. A hardcoded API key that passed code review, a rate limiter that wasn't enabled, a PII scanner that only runs in dev — these are deploy errors, not code errors. The security deployment checklist is the last line of defense.

Deploy checklists for AI systems are different from those of traditional web applications. In addition to the classics (HTTPS, CORS, secrets), you need to verify that the AI defenses are active: is the system prompt hardened? Is the injection detector in the production pipeline? Does the PII scanner process real outputs? These verifications don't exist in generic checklists.

This capsule guides you through creating a 50+ item checklist organized by category, with automated verifications where possible and manual verifications where necessary. At the end you will have an executable DeploymentChecklist that generates a pass/fail report and integrates into your CI/CD pipeline as a pre-deploy gate.

Checklist by category

Secrets & Credentials

API Security

Input/Output Security

PII Protection

Monitoring & Observability

Infrastructure

LLM-Specific

DeploymentChecklist class

The DeploymentChecklist models each item as a pass/fail verification. The automatic ones run programmatically; the manual ones require human confirmation.

from pydantic import BaseModel, Field, computed_field
from enum import Enum
from datetime import datetime
from typing import Optional, Callable

class CheckType(str, Enum):
    AUTOMATIC = "automatic"
    MANUAL = "manual"

class CheckStatus(str, Enum):
    PENDING = "pending"
    PASS = "pass"
    FAIL = "fail"
    SKIP = "skip"

class CheckCategory(str, Enum):
    SECRETS = "secrets"
    API_SECURITY = "api_security"
    INPUT_OUTPUT = "input_output"
    PII = "pii"
    MONITORING = "monitoring"
    INFRASTRUCTURE = "infrastructure"
    LLM_SPECIFIC = "llm_specific"

class ChecklistItem(BaseModel):
    id: str
    category: CheckCategory
    description: str
    check_type: CheckType
    status: CheckStatus = CheckStatus.PENDING
    verification_method: str
    notes: str = ""
    verified_by: str = ""
    verified_at: Optional[datetime] = None
    environments: list[str] = Field(default_factory=lambda: ["staging", "production"])

class ChecklistReport(BaseModel):
    project_name: str
    environment: str
    run_date: datetime
    total_items: int
    passed: int
    failed: int
    pending: int
    pass_rate: float
    deploy_approved: bool
    failed_items: list[dict] = Field(default_factory=list)
    summary: str = ""

class DeploymentChecklist(BaseModel):
    """Checklist with automatic/manual verification and a pre-deploy gate."""
    project_name: str
    environment: str = "production"
    items: list[ChecklistItem] = Field(default_factory=list)
    auto_checks: dict[str, Callable[[], bool]] = Field(default_factory=dict, exclude=True)

    class Config:
        arbitrary_types_allowed = True

    def add_item(self, item: ChecklistItem) -> None:
        self.items.append(item)

    def register_auto_check(self, item_id: str, check_fn: Callable[[], bool]) -> None:
        self.auto_checks[item_id] = check_fn

    def run_automatic_checks(self) -> int:
        """Runs the registered automatic verifications."""
        executed = 0
        for item in self.items:
            if item.check_type != CheckType.AUTOMATIC or item.id not in self.auto_checks:
                continue
            if self.environment not in item.environments:
                item.status = CheckStatus.SKIP
                continue
            try:
                result = self.auto_checks[item.id]()
                item.status = CheckStatus.PASS if result else CheckStatus.FAIL
                item.verified_at = datetime.now()
                item.verified_by = "automated"
                executed += 1
            except Exception as e:
                item.status = CheckStatus.FAIL
                item.notes = f"Check error: {e}"
                executed += 1
        return executed

    def mark_manual(self, item_id: str, passed: bool, verified_by: str, notes: str = "") -> None:
        for item in self.items:
            if item.id == item_id:
                item.status = CheckStatus.PASS if passed else CheckStatus.FAIL
                item.verified_by = verified_by
                item.verified_at = datetime.now()
                item.notes = notes

    def generate_report(self) -> ChecklistReport:
        applicable = [i for i in self.items if self.environment in i.environments]
        total = len(applicable)
        passed = sum(1 for i in applicable if i.status == CheckStatus.PASS)
        failed = sum(1 for i in applicable if i.status == CheckStatus.FAIL)
        pending = sum(1 for i in applicable if i.status == CheckStatus.PENDING)
        pass_rate = (passed / total * 100) if total > 0 else 0
        # Deploy approved only without failures or pending
        deploy_ok = (failed == 0 and pending == 0)
        failed_items = [{"id": i.id, "description": i.description, "notes": i.notes} for i in applicable if i.status == CheckStatus.FAIL]

        summary = (f"✅ DEPLOY APPROVED — all {total} checks passed" if deploy_ok
                   else f"❌ DEPLOY BLOCKED — {failed} failed" if failed > 0
                   else f"⏳ DEPLOY PENDING — {pending} not verified")

        return ChecklistReport(
            project_name=self.project_name, environment=self.environment,
            run_date=datetime.now(), total_items=total, passed=passed,
            failed=failed, pending=pending, pass_rate=pass_rate,
            deploy_approved=deploy_ok, failed_items=failed_items, summary=summary,
        )

    def print_report(self) -> None:
        r = self.generate_report()
        print(f"{'='*55}")
        print(f"Deployment Checklist: {r.project_name} ({r.environment})")
        print(f"{'='*55}")
        print(f"Total: {r.total_items} | Pass: {r.passed} | Fail: {r.failed} | Pending: {r.pending}")
        print(f"Pass rate: {r.pass_rate:.0f}%\n{r.summary}")
        if r.failed_items:
            print("\nFailed:")
            for item in r.failed_items:
                print(f"  ❌ {item['id']}: {item['description']}")

Automated verification

Automatic verifications remove human error from the most critical checks.

import os
import subprocess
from pathlib import Path
import re

class AutomatedSecurityChecks:
    """Automated security configuration checks."""

    def __init__(self, project_root: str = "."):
        self.root = Path(project_root)

    def check_no_env_committed(self) -> bool:
        """Verifies that .env is not tracked by git."""
        gitignore = self.root / ".gitignore"
        if not gitignore.exists():
            return False
        content = gitignore.read_text()
        env_ignored = any(
            line.strip() in (".env", ".env*", "*.env")
            for line in content.splitlines() if not line.strip().startswith("#")
        )
        if not env_ignored:
            return False
        try:
            result = subprocess.run(
                ["git", "ls-files", ".env"], capture_output=True, text=True, cwd=self.root,
            )
            return len(result.stdout.strip()) == 0
        except FileNotFoundError:
            return True

    def check_no_secrets_in_code(self) -> bool:
        """Looks for hardcoded secrets. Returns True if it finds none."""
        patterns = [
            r"sk-[a-zA-Z0-9]{20,}",
            r"AKIA[0-9A-Z]{16}",
            r"(?i)password\s*=\s*['\"][^'\"]+['\"]",
        ]
        skip_dirs = {"venv", "node_modules", ".git", "__pycache__"}
        for py_file in self.root.rglob("*.py"):
            if any(d in py_file.parts for d in skip_dirs):
                continue
            try:
                content = py_file.read_text(errors="ignore")
                for p in patterns:
                    if re.search(p, content):
                        print(f"  ⚠ Secret in: {py_file.relative_to(self.root)}")
                        return False
            except (PermissionError, OSError):
                continue
        return True

    def check_rate_limiting(self) -> bool:
        """Verifies that rate limiting configuration exists."""
        indicators = ["rate_limit", "ratelimit", "throttle", "RATE_LIMIT"]
        for f in self.root.rglob("*.py"):
            if any(d in f.parts for d in ("venv", ".git")):
                continue
            try:
                content = f.read_text(errors="ignore")
                if any(ind in content for ind in indicators):
                    return True
            except (PermissionError, OSError):
                continue
        return False

    def check_pii_scanner_active(self) -> bool:
        """Verifies that the PII scanner is in the pipeline."""
        indicators = ["PIIScanner", "pii_scan", "presidio", "PostLLMPIIScanner"]
        for f in self.root.rglob("*.py"):
            if any(d in f.parts for d in ("venv", ".git", "test")):
                continue
            try:
                if any(ind in f.read_text(errors="ignore") for ind in indicators):
                    return True
            except (PermissionError, OSError):
                continue
        return False

    def run_all(self) -> dict[str, bool]:
        checks = {
            "no_env_committed": self.check_no_env_committed,
            "no_secrets_in_code": self.check_no_secrets_in_code,
            "rate_limiting": self.check_rate_limiting,
            "pii_scanner_active": self.check_pii_scanner_active,
        }
        results = {}
        for name, fn in checks.items():
            try:
                results[name] = fn()
            except Exception as e:
                print(f"  ⚠ '{name}' error: {e}")
                results[name] = False
        return results

checker = AutomatedSecurityChecks(".")
print("=== Automated Security Checks ===\n")
for name, passed in checker.run_all().items():
    print(f"{'✅' if passed else '❌'} {name}")

Manual verification

Some items require human judgment. They are documented with clear criteria for the reviewer.

from pydantic import BaseModel, Field
from datetime import datetime

class ManualVerification(BaseModel):
    item_id: str
    description: str
    reviewer: str
    review_date: datetime
    passed: bool
    evidence: str
    notes: str = ""

class ManualCheckRegistry(BaseModel):
    """Registry of manual verifications with accountability."""
    verifications: list[ManualVerification] = Field(default_factory=list)

    def record(self, item_id: str, description: str, reviewer: str, passed: bool, evidence: str, notes: str = "") -> None:
        self.verifications.append(ManualVerification(
            item_id=item_id, description=description, reviewer=reviewer,
            review_date=datetime.now(), passed=passed, evidence=evidence, notes=notes,
        ))

    def pending(self, all_ids: list[str]) -> list[str]:
        verified = {v.item_id for v in self.verifications}
        return [mid for mid in all_ids if mid not in verified]

    def summary(self) -> str:
        lines = ["=== Manual Verifications ===", ""]
        for v in self.verifications:
            icon = "✅" if v.passed else "❌"
            lines.append(f"{icon} {v.item_id}: {v.description}")
            lines.append(f"   By {v.reviewer} on {v.review_date.strftime('%Y-%m-%d')}")
            lines.append(f"   Evidence: {v.evidence}")
            if v.notes:
                lines.append(f"   Notes: {v.notes}")
        return "\n".join(lines)

registry = ManualCheckRegistry()
registry.record("IO-02", "System prompt hardened", "Ana García", True, "Prompt v2.3 reviewed")
registry.record("M-07", "Incident response runbook", "Diana Ruiz", False, "Missing weekend escalation contacts", "Update by 2026-03-20")
print(registry.summary())

Pre-launch vs Post-launch checks

from pydantic import BaseModel
from enum import Enum

class CheckPhase(str, Enum):
    PRE_LAUNCH = "pre_launch"
    POST_LAUNCH = "post_launch"
    BOTH = "both"

class PhaseCheck(BaseModel):
    id: str
    description: str
    phase: CheckPhase
    blocks_deploy: bool

checks = [
    PhaseCheck(id="PRE-01", description="Adversarial test suite passes", phase=CheckPhase.PRE_LAUNCH, blocks_deploy=True),
    PhaseCheck(id="PRE-02", description="No secrets in source code", phase=CheckPhase.PRE_LAUNCH, blocks_deploy=True),
    PhaseCheck(id="PRE-03", description="PII scanner tests pass", phase=CheckPhase.PRE_LAUNCH, blocks_deploy=True),
    PhaseCheck(id="POST-01", description="No PII in production logs", phase=CheckPhase.POST_LAUNCH, blocks_deploy=False),
    PhaseCheck(id="POST-02", description="Rate limiting works under load", phase=CheckPhase.POST_LAUNCH, blocks_deploy=False),
    PhaseCheck(id="POST-03", description="Injection alerts fire correctly", phase=CheckPhase.POST_LAUNCH, blocks_deploy=False),
    PhaseCheck(id="BOTH-01", description="Health check returns 200", phase=CheckPhase.BOTH, blocks_deploy=True),
    PhaseCheck(id="BOTH-02", description="Error rate < 5%", phase=CheckPhase.BOTH, blocks_deploy=True),
]

for phase in CheckPhase:
    phase_checks = [c for c in checks if c.phase == phase]
    print(f"\n[{phase.value.upper()}]")
    for c in phase_checks:
        tag = "🚫 BLOCKER" if c.blocks_deploy else "📋 Advisory"
        print(f"  {c.id}: {c.description} — {tag}")

Checklist per environment

Not all checks apply to all environments.

from pydantic import BaseModel, Field

class EnvironmentConfig(BaseModel):
    name: str
    required_checks: list[str]
    optional_checks: list[str]
    strict_mode: bool

def build_env_configs() -> dict[str, EnvironmentConfig]:
    return {
        "development": EnvironmentConfig(
            name="Development", strict_mode=False,
            required_checks=["S-01", "IO-01", "P-01"],
            optional_checks=["A-03", "M-01"],
        ),
        "staging": EnvironmentConfig(
            name="Staging", strict_mode=True,
            required_checks=["S-01", "S-02", "S-06", "A-01", "A-02", "A-03", "A-04", "IO-01", "IO-03", "P-01", "P-02", "P-03", "L-01", "L-07"],
            optional_checks=["M-02", "M-04", "I-03"],
        ),
        "production": EnvironmentConfig(
            name="Production", strict_mode=True,
            required_checks=[
                "S-01", "S-02", "S-03", "S-04", "S-05", "S-06", "S-07",
                "A-01", "A-02", "A-03", "A-04", "A-05", "A-06", "A-07",
                "IO-01", "IO-02", "IO-03", "IO-04", "IO-05", "IO-06", "IO-07",
                "P-01", "P-02", "P-03", "P-04", "P-05", "P-06", "P-07",
                "M-01", "M-02", "M-03", "M-04", "M-05", "M-06", "M-07",
                "I-01", "I-02", "I-03", "I-04", "I-05", "I-06", "I-07",
                "L-01", "L-02", "L-03", "L-04", "L-05", "L-06", "L-07", "L-08",
            ],
            optional_checks=[],
        ),
    }

for name, cfg in build_env_configs().items():
    print(f"{cfg.name}: {len(cfg.required_checks)} required, {len(cfg.optional_checks)} optional, strict={'ON' if cfg.strict_mode else 'OFF'}")

# Expected output:
# Development: 3 required, 2 optional, strict=OFF
# Staging: 14 required, 3 optional, strict=ON
# Production: 50 required, 0 optional, strict=ON

CI/CD integration

The checklist runs automatically in GitHub Actions as a pre-deploy gate.

# .github/workflows/security-checklist.yml
name: Security Deployment Checklist

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  security-checklist:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: pip install pip-audit pydantic

      - name: Check for secrets in code
        run: |
          if grep -rn "sk-[a-zA-Z0-9]\{20,\}" --include="*.py" .; then
            echo "::error::Hardcoded API keys found"
            exit 1
          fi

      - name: Check .env not tracked
        run: |
          if git ls-files .env | grep -q ".env"; then
            echo "::error::.env tracked by git"
            exit 1
          fi

      - name: Dependency vulnerability scan
        run: pip-audit || true

      - name: Run adversarial test suite
        run: python -m pytest tests/security/ -v --tb=short

      - name: Run deployment checklist
        run: python scripts/run_deploy_checklist.py --env production

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: security-checklist-report
          path: checklist-report.json

The script the workflow runs:

import sys
import json
from datetime import datetime
from pathlib import Path

def run_deploy_checklist(environment: str = "production") -> bool:
    """Runs the checklist and returns True if it passes."""
    print(f"Running checklist for: {environment}")
    results: dict[str, bool] = {}
    failures: list[str] = []

    # .gitignore includes .env
    gitignore = Path(".gitignore")
    check = gitignore.exists() and ".env" in gitignore.read_text()
    results["env_in_gitignore"] = check
    if not check:
        failures.append("S-01: .env not in .gitignore")

    # Security tests exist
    check = len(list(Path(".").rglob("tests/security/*.py"))) > 0
    results["security_tests_exist"] = check
    if not check:
        failures.append("L-07: No security tests found")

    # System prompt versioned
    prompt_files = list(Path(".").rglob("*system_prompt*")) + list(Path(".").rglob("prompts/*.txt"))
    check = len(prompt_files) > 0
    results["system_prompt_versioned"] = check
    if not check:
        failures.append("L-05: No system prompt file found")

    # Save report
    report = {
        "environment": environment,
        "timestamp": datetime.now().isoformat(),
        "results": results,
        "failures": failures,
        "passed": len(failures) == 0,
    }
    with open("checklist-report.json", "w") as f:
        json.dump(report, f, indent=2)

    for name, ok in results.items():
        print(f"{'✅' if ok else '❌'} {name}")

    if failures:
        print(f"\n❌ BLOCKED — Fix: {'; '.join(failures)}")
        return False
    print("\n✅ DEPLOY APPROVED")
    return True

if __name__ == "__main__":
    env = "production"
    if "--env" in sys.argv:
        idx = sys.argv.index("--env")
        env = sys.argv[idx + 1] if idx + 1 < len(sys.argv) else "production"
    sys.exit(0 if run_deploy_checklist(env) else 1)

Troubleshooting

1. The automatic checklist passes locally but fails in CI

Problem: The checks depend on files or configurations that exist locally but not in CI.

Solution: Ensure that everything the checks need is in the repository or is generated in the pipeline. Use CI environment variables for environment configurations. For checks that require external services (vault, cloud), use mocks in CI.

2. False positives in the secret scanner

Problem: The scanner detects strings that look like API keys but are test fixtures or legitimate hashes.

Solution: Keep a .secret-scan-ignore with documented exceptions. Refine the regex for provider-specific prefixes. Never disable the scanner entirely because of false positives.

3. Manual checks get skipped due to time pressure

Problem: The team marks checks as "passed" without verifying because there's pressure to deploy.

Solution: Add accountability: each manual check requires a reviewer and specific evidence. Highlight items without evidence. Implement a 30-day expiration for re-verification.

4. The production checklist has too many items

Problem: With 50+ items, the team loses motivation to complete it.

Solution: Automate everything possible — target: ≤ 15 manual checks. Group by owner so nobody sees all 50, only their own. After the first 3 deploys, only re-verify items that changed.

5. The CI pipeline takes too long due to security checks

Problem: The checks add 10+ minutes to the pipeline.

Solution: Parallelize independent checks in separate jobs. Cache pip-audit and re-run only when requirements.txt changes. Run the full adversarial suite only on merges to main; a subset on PRs. Target: < 5 minutes of overhead.

Exercises

Exercise 1: Build a complete checklist with auto-checks

Implement a DeploymentChecklist with 15+ items from 4 categories. Register auto-checks for 5+ items. Run the auto-checks, mark 3 manual, generate a report.

from datetime import datetime

checklist = DeploymentChecklist(project_name="AI Assistant v2", environment="production")

items_def = [
    ("S-01", CheckCategory.SECRETS, "No .env committed", CheckType.AUTOMATIC, "git check"),
    ("S-02", CheckCategory.SECRETS, "Keys in vault", CheckType.MANUAL, "Vault health"),
    ("S-06", CheckCategory.SECRETS, "No secrets in logs", CheckType.AUTOMATIC, "Log scan"),
    ("A-01", CheckCategory.API_SECURITY, "HTTPS enforced", CheckType.AUTOMATIC, "curl check"),
    ("A-03", CheckCategory.API_SECURITY, "Rate limiting active", CheckType.AUTOMATIC, "Config check"),
    ("A-04", CheckCategory.API_SECURITY, "Auth required", CheckType.AUTOMATIC, "401 test"),
    ("IO-01", CheckCategory.INPUT_OUTPUT, "Injection detector active", CheckType.AUTOMATIC, "Payload test"),
    ("IO-02", CheckCategory.INPUT_OUTPUT, "System prompt hardened", CheckType.MANUAL, "Prompt review"),
    ("IO-03", CheckCategory.INPUT_OUTPUT, "Output filter active", CheckType.AUTOMATIC, "Output test"),
    ("P-01", CheckCategory.PII, "PII scanner input", CheckType.AUTOMATIC, "PII test"),
    ("P-02", CheckCategory.PII, "PII scanner output", CheckType.AUTOMATIC, "PII test"),
    ("P-03", CheckCategory.PII, "Logs no PII", CheckType.MANUAL, "Log sample"),
    ("M-01", CheckCategory.MONITORING, "Structured logging", CheckType.AUTOMATIC, "Format check"),
    ("L-01", CheckCategory.LLM_SPECIFIC, "Model pinned", CheckType.AUTOMATIC, "Config check"),
    ("L-06", CheckCategory.LLM_SPECIFIC, "OWASP updated", CheckType.MANUAL, "Doc review"),
]

for id_, cat, desc, ctype, verif in items_def:
    checklist.add_item(ChecklistItem(id=id_, category=cat, description=desc, check_type=ctype, verification_method=verif))

# Register auto-checks (simulated)
for id_, fn in [("S-01", lambda: True), ("S-06", lambda: True), ("A-01", lambda: True),
                ("A-03", lambda: False), ("A-04", lambda: True), ("IO-01", lambda: True),
                ("IO-03", lambda: True), ("P-01", lambda: True), ("P-02", lambda: True),
                ("M-01", lambda: True), ("L-01", lambda: True)]:
    checklist.register_auto_check(id_, fn)

print(f"Executed {checklist.run_automatic_checks()} auto checks\n")

checklist.mark_manual("S-02", True, "Ana García", "Vault verified 2026-03-10")
checklist.mark_manual("IO-02", True, "Carlos López", "Prompt v2.3 reviewed")
checklist.mark_manual("P-03", False, "Diana Ruiz", "Found emails in 3 log entries")

checklist.print_report()

# Expected output:
# =======================================================
# Deployment Checklist: AI Assistant v2 (production)
# =======================================================
# Total: 15 | Pass: 12 | Fail: 2 | Pending: 1
# Pass rate: 80%
# ❌ DEPLOY BLOCKED — 2 failed
# Failed:
#   ❌ A-03: Rate limiting active
#   ❌ P-03: Logs no PII

Exercise 2: Create an environment-aware runner

Implement a runner that filters the checklist by environment. Development: only required. Staging: required + optional advisory. Production: everything mandatory.

from pydantic import BaseModel, Field

class EnvironmentRunner(BaseModel):
    configs: dict[str, EnvironmentConfig] = Field(default_factory=dict)

    def run_for(self, env: str, items: list[ChecklistItem]) -> dict:
        cfg = self.configs.get(env)
        if not cfg:
            return {"error": f"Unknown: {env}"}
        required = [i for i in items if i.id in cfg.required_checks]
        optional = [i for i in items if i.id in cfg.optional_checks]
        req_fail = sum(1 for i in required if i.status.value == "fail")
        deploy_ok = req_fail == 0 if cfg.strict_mode else req_fail <= 1
        return {
            "environment": env,
            "required": f"{sum(1 for i in required if i.status.value == 'pass')}/{len(required)}",
            "optional": f"{sum(1 for i in optional if i.status.value == 'pass')}/{len(optional)}",
            "deploy": deploy_ok,
            "failed": [i.id for i in required if i.status.value == "fail"],
        }

    def compare(self, items: list[ChecklistItem]) -> str:
        lines = ["=== Environment Comparison ==="]
        for env in self.configs:
            r = self.run_for(env, items)
            icon = "✅" if r["deploy"] else "❌"
            lines.append(f"{icon} {env.upper()}: {r['required']} required, {r['optional']} optional")
            if r["failed"]:
                lines.append(f"   Failed: {', '.join(r['failed'])}")
        return "\n".join(lines)

runner = EnvironmentRunner(configs=build_env_configs())

# Simulate items with A-03 and P-03 failing
sample = []
for id_, cat, desc, ctype, verif in items_def:
    item = ChecklistItem(id=id_, category=cat, description=desc, check_type=ctype, verification_method=verif)
    item.status = CheckStatus.FAIL if id_ in ("A-03", "P-03") else CheckStatus.PASS
    sample.append(item)

print(runner.compare(sample))

Exercise 3: Implement a checklist diff between deploys

Create a function that compares the checklist between two deploys and shows what improved, regressed, or stayed the same.

from pydantic import BaseModel, Field
from datetime import datetime

class DeploySnapshot(BaseModel):
    deploy_id: str
    date: datetime
    results: dict[str, str]  # item_id → status

def checklist_diff(before: DeploySnapshot, after: DeploySnapshot) -> dict:
    improved, regressed, unchanged = [], [], []
    for fid, old in before.results.items():
        new = after.results.get(fid, "removed")
        if old == new:
            unchanged.append(fid)
        elif old == "fail" and new == "pass":
            improved.append(fid)
        elif old == "pass" and new == "fail":
            regressed.append(fid)
    new_checks = [fid for fid in after.results if fid not in before.results]
    return {"improved": improved, "regressed": regressed, "unchanged": unchanged, "new": new_checks}

v1 = DeploySnapshot(deploy_id="v2.0", date=datetime(2026, 3, 1), results={"S-01": "pass", "A-03": "fail", "P-03": "fail", "IO-01": "pass"})
v2 = DeploySnapshot(deploy_id="v2.1", date=datetime(2026, 3, 14), results={"S-01": "pass", "A-03": "pass", "P-03": "pass", "IO-01": "pass", "L-01": "pass"})

d = checklist_diff(v1, v2)
print(f"v2.0 → v2.1:")
print(f"  📈 Improved: {d['improved']}")
print(f"  📉 Regressed: {d['regressed']}")
print(f"  ➡️  Unchanged: {d['unchanged']}")
print(f"  🆕 New: {d['new']}")

Exercise 4: Create a Slack notifier for the checklist

Implement a ChecklistNotifier that generates a Slack payload with status, failures, and a call-to-action.

import json
from pydantic import BaseModel, Field
from datetime import datetime

class ChecklistNotifier(BaseModel):
    webhook_url: str
    channel: str = "#deploys"
    mention_on_failure: list[str] = Field(default_factory=lambda: ["@security-team"])

    def build_payload(self, report: ChecklistReport) -> dict:
        emoji = "✅" if report.deploy_approved else "🚨"
        status = "APPROVED" if report.deploy_approved else "BLOCKED"
        color = "#36a64f" if report.deploy_approved else "#cc0000"

        header = f"{emoji} Deploy Checklist: *{status}*"
        if not report.deploy_approved:
            header += f"\n{' '.join(self.mention_on_failure)}"

        stats = (f"*Project:* {report.project_name} | *Env:* {report.environment}\n"
                 f"*Pass rate:* {report.pass_rate:.0f}% ({report.passed}/{report.total_items})")

        blocks = [
            {"type": "section", "text": {"type": "mrkdwn", "text": header}},
            {"type": "section", "text": {"type": "mrkdwn", "text": stats}},
        ]

        if report.failed_items:
            fail_text = "*Failed:*\n" + "\n".join(f"• `{f['id']}` {f['description']}" for f in report.failed_items)
            blocks.append({"type": "section", "text": {"type": "mrkdwn", "text": fail_text}})

        if not report.deploy_approved:
            blocks.append({"type": "section", "text": {"type": "mrkdwn", "text": "⚡ Fix failed checks and re-run pipeline."}})

        return {"channel": self.channel, "attachments": [{"color": color, "blocks": blocks}]}

notifier = ChecklistNotifier(webhook_url="https://hooks.slack.com/services/T00/B00/X")
sample_report = ChecklistReport(
    project_name="AI Assistant v2", environment="production",
    run_date=datetime.now(), total_items=15, passed=13, failed=2,
    pending=0, skipped=0, pass_rate=86.7, deploy_approved=False,
    failed_items=[{"id": "A-03", "description": "Rate limiting", "notes": ""}, {"id": "P-03", "description": "PII in logs", "notes": ""}],
    summary="BLOCKED",
)
print(json.dumps(notifier.build_payload(sample_report), indent=2, ensure_ascii=False))

Summary

• 🔒 A deployment checklist for AI includes verifications of the system prompt, injection defense, PII scanning, and model configuration — not just HTTPS and CORS
• 📋 The 50+ items are organized into 7 categories: Secrets, API Security, Input/Output, PII, Monitoring, Infrastructure, LLM-Specific
• ⚙️ Automatic verifications remove human error: .env not committed, no secrets in code, rate limiting, PII scanner active
• 👤 Manual verifications require documented evidence: who verified, when, and with what proof
• 🔄 Pre-launch checks block the deploy; post-launch checks validate in production without blocking but require a response
• 🏗️ The per-environment configuration scales the rigor: development 3 checks, staging 14, production 50
• 🤖 The CI/CD integration with GitHub Actions runs checks automatically as a pre-deploy gate on every push to main
• 📱 Slack notifications give immediate visibility of failures to the security team

Next capsule: In capsule 05 you will document your system's security decisions and create a professional incident response runbook.

Additional resources

1. OWASP AI Security Deployment Guide — Deployment guidelines specific to AI
2. CIS Benchmarks — Infrastructure security benchmarks
3. GitHub Actions Security Hardening — Security for CI/CD workflows
4. pip-audit — Python Dependency Scanner — Vulnerability scanner for Python dependencies
5. Trivy — Container Security Scanner — Scanner for Docker images and IaC
6. NIST SP 800-53 — Security Controls — Reference catalog of security controls
7. Drata — Compliance Automation — Compliance and checklist automation platform

Created: March 2026 Version: 1.0