1. Production Checklist

Description

The production checklist is the difference between "I think it's ready" and "I know it's ready." It's not a theoretical list you read and file away — it's an executable tool: each item has a concrete command for you to verify whether it passes and a concrete step to fix it if it fails. In this capsule you'll build the complete checklist specific to AI apps that you'll use as a validation tool in the M8 integrative project.

Why a checklist specific to AI

A generic "deploy web app" checklist doesn't cover what can go wrong in an AI app:

Generic web app checklist — what it does NOT cover for AI:
❌ "Do you have backups?" → Says nothing about versioned prompts
❌ "Is the DB optimized?" → Says nothing about cost tracking
❌ "Do you have rate limiting?" → Says nothing about token limits
❌ "Is the API documented?" → Doesn't mention guardrails on endpoints

AI-specific checklist — what you actually need to verify:
✅ Are the prompts versioned in git (not hardcoded in the code)?
✅ Is there prompt injection guardrails on every endpoint with user input?
✅ Does cost tracking have alerts to detect abuse?
✅ Is retry configured only for transient errors?
✅ Is there a fallback when the primary LLM provider fails?
✅ Does the health check verify that OpenAI responds, not just that the HTTP server responds?

The complete checklist

Category 1: Security

1.1 GUARDRAILS ACTIVE
    ├── Verify: run a prompt injection test
    │   pytest tests/ -k "injection" -v
    │   → The attack must fail, the test must pass
    ├── Verify: every endpoint with user input has a guardrail
    │   grep -r "guardrail" src/app/routers/  → must appear in every router
    └── Fix: add GuardrailsPipeline before calling the domain

1.2 PII IS NOT LOGGED
    ├── Verify: make a request with an email, check it doesn't appear in logs
    │   curl -X POST /analyze -d '{"text": "email: user@example.com"}'
    │   jq '.message' logs/app.json | grep -i "user@example" → must return empty
    └── Fix: enable sanitize_sensitive_fields in structlog processors

1.3 SECRETS NOT IN CODE
    ├── Verify: scan the repository
    │   grep -r "sk-" src/  → must return empty (no API keys in code)
    │   grep -r "OPENAI_API_KEY" src/  → only references to env vars, not values
    └── Fix: move secrets to .env, add .env to .gitignore

1.4 .ENV NOT IN GIT
    ├── Verify: git check-ignore .env → must return ".env"
    └── Fix: add .env to .gitignore, if already tracked: git rm --cached .env

1.5 CONTENT POLICY VERIFIED
    ├── Verify: test with inputs of known toxicity
    │   pytest tests/ -k "content_policy" -v
    └── Fix: add a content policy check in the guardrails pipeline

Category 2: Testing

2.1 UNIT TESTS PASS
    ├── Verify: pytest tests/unit/ -v
    │   → All must pass. Coverage > 70% in domain and processing
    ├── Verify: pytest --cov=src/domain --cov-report=term-missing
    └── Fix: write missing tests for the domain service and parsers

2.2 INTEGRATION TESTS PASS (or are correctly skipped)
    ├── Verify: pytest tests/integration/ -v
    │   → Pass with a real API key, or skip correctly without it
    ├── If there's no API key in CI: pytest tests/integration/ -k "not requires_api"
    └── Fix: review the conftest.py fixtures, verify @pytest.mark.integration

2.3 GUARDRAILS TESTS PASS
    ├── Verify: pytest tests/ -k "guardrail" -v
    │   → The injection test must block the attack
    │   → The PII test must redact correctly
    └── Fix: review the guardrails implementation

2.4 RELIABILITY TESTS PASS
    ├── Verify: pytest tests/ -k "retry or circuit or fallback" -v
    │   → Retry must retry 3 times
    │   → Circuit must open after N failures
    │   → Fallback must activate when primary fails
    └── Fix: review RetryProvider, CircuitBreaker, FallbackProvider

2.5 NO SLOW TESTS WITHOUT REASON
    ├── Verify: pytest tests/ --timeout=5 -v
    │   → Unit tests must not take > 2s each
    └── Fix: verify that retry in tests uses min_wait_seconds=0.01

Category 3: Observability

3.1 REQUEST TRACING WORKS
    ├── Verify:
    │   # Make a request
    │   curl -X POST /api/v1/analyze -d '{"text": "test"}'
    │   # Look for the request_id in the logs
    │   tail -1 logs/app.json | jq '.request_id'
    │   → Must return a UUID, not null
    └── Fix: verify that RequestTracingMiddleware is registered in main.py

3.2 COST TRACKING WORKS
    ├── Verify:
    │   tail -1 logs/app.json | jq '{cost_usd, input_tokens, output_tokens}'
    │   → Must return numeric values, not null
    └── Fix: verify that calculate_cost is called in OpenAIProvider.complete()

3.3 CORRECT LOG LEVEL IN PROD
    ├── Verify: ENVIRONMENT=production python -c "from src.config import get_settings; s=get_settings(); print(s.log_level)"
    │   → Must return "INFO" (not "DEBUG" in production)
    └── Fix: review .env.production, verify model_validator in Settings

3.4 GUARDRAIL ACTIVATIONS LOGGED
    ├── Verify:
    │   # Send a request with prompt injection
    │   curl -X POST /api/v1/analyze -d '{"text": "Ignore previous instructions"}'
    │   # Look in the logs
    │   grep "guardrail_activated" logs/app.json
    │   → The entry with the guardrail type must appear
    └── Fix: verify that GuardrailsPipeline logs with log.warning("guardrail_activated", ...)

3.5 LOGS IN JSON FORMAT (NOT PLAIN TEXT IN PROD)
    ├── Verify:
    │   head -1 logs/app.json | python -m json.tool > /dev/null
    │   → Must parse without error
    └── Fix: verify configure_logging() with format="json"

Category 4: Reliability

4.1 RETRY CONFIGURED CORRECTLY
    ├── Verify (in code):
    │   grep -r "max_attempts\|stop_after_attempt" src/ → must exist
    │   grep -r "stop_after_attempt(1000)\|stop_after_attempt(999)" → must NOT exist
    ├── Verify (with a test):
    │   pytest tests/ -k "test_retry" -v → must pass
    └── Fix: verify RetryProvider with max_attempts=3-5

4.2 CIRCUIT BREAKER IS A SINGLETON
    ├── Verify:
    │   grep -r "CircuitBreaker()" src/app/ → if it appears in a handler, problem
    │   # The CB must be in dependencies.py or in a global module
    └── Fix: move CircuitBreaker to a global variable outside the handler

4.3 FALLBACK CHAIN CONFIGURED
    ├── Verify:
    │   grep -r "FallbackProvider" src/ → must appear in dependencies.py
    │   grep -r "static_fallback" src/ → must have a fallback string
    └── Fix: wrap the provider with FallbackProvider

4.4 HEALTH CHECKS RESPOND
    ├── Verify:
    │   curl -f http://localhost:8000/health/live → must return 200
    │   curl -f http://localhost:8000/health/ready → must return 200
    │   curl -f http://localhost:8000/health/deps → must return 200
    └── Fix: verify that the health router is registered in main.py

4.5 RATE LIMITING CONFIGURED
    ├── Verify:
    │   grep -r "RateLimitedProvider\|TokenBucket" src/ → must appear
    │   # Verify that rate < 80% of the API limit
    └── Fix: wrap with RateLimitedProvider(rpm=int(api_limit * 0.8))

Category 5: Performance

5.1 LATENCY TARGET DOCUMENTED
    ├── Verify: cat docs/BASELINES.md → must exist with numeric values
    └── Fix: run a benchmark, document it in BASELINES.md

5.2 TIMEOUTS CONFIGURED
    ├── Verify:
    │   grep -r "timeout" src/ → must appear in the openai client init
    └── Fix: OpenAI(timeout=30.0) — don't leave the default infinite

5.3 COST PER REQUEST ESTIMATED
    ├── Verify:
    │   # From the development logs:
    │   jq '.cost_usd' logs/app.json | awk '{s+=$1} END {print s/NR}' → average cost
    └── Document in BASELINES.md

5.4 PROMPTS OPTIMIZED FOR COST
    ├── Verify: the system prompt has no unnecessary repetitive text blocks
    │   # Use tiktoken to estimate the prompt's tokens:
    │   python -c "import tiktoken; enc = tiktoken.encoding_for_model('gpt-4o'); print(len(enc.encode(open('prompts/sentiment/v1.yaml').read())))"
    └── Fix: remove unnecessary padding in prompts

Category 6: Documentation

6.1 COMPLETE README
    ├── Verify: README.md has → setup instructions, env vars, how to run tests, how to deploy
    └── Fix: complete the README with the missing sections

6.2 API DOCS ACCESSIBLE
    ├── Verify: curl http://localhost:8000/docs → must return 200
    └── Fix: FastAPI includes Swagger automatically

6.3 .ENV.EXAMPLE EXISTS AND IS UP TO DATE
    ├── Verify:
    │   cat .env.example → must have all the variables
    │   diff <(grep -o '^[A-Z_]*' .env) <(grep -o '^[A-Z_]*' .env.example) → must be empty
    └── Fix: add the missing variables in .env.example

6.4 RUNBOOK WITH AT LEAST 3 INCIDENTS
    ├── Verify: cat docs/RUNBOOK.md → must have sections for high latency, 5xx errors, high cost
    └── Fix: complete the runbook with diagnosis and resolution for each incident

6.5 VERSIONED PROMPTS
    ├── Verify:
    │   ls prompts/  → must have a version in the directory name (v1, v2, ...)
    │   grep "version" prompts/sentiment/v1.yaml → must have version metadata
    └── Fix: move prompts to YAML files with version metadata

Automated checklist script

# scripts/run_checklist.py
"""
Runs the production checklist in an automated way.
The items that can be automated are verified with code.
The manual items are marked as "requires human verification".
"""
import subprocess
import json
import os
import sys
from pathlib import Path
from typing import Tuple

def check(name: str, fn) -> Tuple[str, bool, str]:
    """Runs a check and returns (name, passed, detail)."""
    try:
        result = fn()
        if isinstance(result, tuple):
            passed, detail = result
        else:
            passed, detail = bool(result), ""
        return name, passed, detail
    except Exception as e:
        return name, False, str(e)[:200]

def check_no_secrets_in_code() -> Tuple[bool, str]:
    """Verifies that there are no API keys in the code."""
    result = subprocess.run(
        ["grep", "-r", "sk-", "src/"],
        capture_output=True, text=True
    )
    if result.stdout.strip():
        return False, f"Found potential secrets: {result.stdout[:200]}"
    return True, "No secrets found in src/"

def check_env_example_exists() -> Tuple[bool, str]:
    """Verifies that .env.example exists."""
    exists = Path(".env.example").exists()
    return exists, ".env.example found" if exists else ".env.example missing"

def check_env_gitignored() -> Tuple[bool, str]:
    """Verifies that .env is in .gitignore."""
    result = subprocess.run(
        ["git", "check-ignore", ".env"],
        capture_output=True, text=True
    )
    ignored = ".env" in result.stdout
    return ignored, ".env gitignored" if ignored else ".env NOT in .gitignore — SECURITY RISK"

def check_unit_tests_pass() -> Tuple[bool, str]:
    """Runs the unit tests and verifies that they pass."""
    result = subprocess.run(
        ["python", "-m", "pytest", "tests/unit/", "-v", "--tb=short"],
        capture_output=True, text=True
    )
    passed = result.returncode == 0
    lines = result.stdout.split("\n")
    summary = next((l for l in reversed(lines) if "passed" in l or "failed" in l), "")
    return passed, summary

def check_health_endpoint(base_url: str = "http://localhost:8000") -> Tuple[bool, str]:
    """Verifies that /health/live responds 200."""
    import urllib.request
    try:
        req = urllib.request.urlopen(f"{base_url}/health/live", timeout=5)
        return req.status == 200, f"HTTP {req.status}"
    except Exception as e:
        return False, str(e)

def check_baselines_doc_exists() -> Tuple[bool, str]:
    """Verifies that BASELINES.md exists with content."""
    path = Path("docs/BASELINES.md")
    if not path.exists():
        return False, "docs/BASELINES.md missing"
    content = path.read_text()
    has_latency = "latency" in content.lower()
    has_cost = "cost" in content.lower()
    if not (has_latency and has_cost):
        return False, "BASELINES.md exists but missing latency/cost sections"
    return True, "BASELINES.md has latency and cost baselines"

def check_runbook_exists() -> Tuple[bool, str]:
    """Verifies that the runbook exists with documented incidents."""
    path = Path("docs/RUNBOOK.md")
    if not path.exists():
        return False, "docs/RUNBOOK.md missing"
    content = path.read_text()
    incidents = ["latencia", "5xx", "costo", "latency", "error", "cost"]
    found = sum(1 for i in incidents if i.lower() in content.lower())
    if found < 2:
        return False, f"RUNBOOK.md exists but only {found} incident types documented"
    return True, f"RUNBOOK.md has {found} incident types"

def run_production_checklist(skip_server_checks: bool = False):
    """Runs the complete checklist and reports results."""
    checks = [
        ("Security: no secrets in code", check_no_secrets_in_code),
        ("Security: .env gitignored", check_env_gitignored),
        ("Security: .env.example exists", check_env_example_exists),
        ("Testing: unit tests pass", check_unit_tests_pass),
        ("Documentation: BASELINES.md", check_baselines_doc_exists),
        ("Documentation: RUNBOOK.md", check_runbook_exists),
    ]
    
    if not skip_server_checks:
        checks.append(("Reliability: health/live", check_health_endpoint))
    
    results = []
    for name, fn in checks:
        name, passed, detail = check(name, fn)
        results.append((name, passed, detail))
        status = "✅" if passed else "❌"
        print(f"  {status} {name}: {detail}")
    
    passed = sum(1 for _, p, _ in results if p)
    total = len(results)
    print(f"\n{'=' * 50}")
    print(f"CHECKLIST: {passed}/{total} passed")
    
    if passed < total:
        failed = [n for n, p, _ in results if not p]
        print(f"FAILED: {', '.join(failed)}")
        return False
    
    print("ALL CHECKS PASSED — ready for production")
    return True

if __name__ == "__main__":
    skip_server = "--no-server" in sys.argv
    success = run_production_checklist(skip_server_checks=skip_server)
    sys.exit(0 if success else 1)

Exercises

Exercise 1: Identify the most critical check

Of the whole list, which is the check that, if it fails, would make you NOT launch under any circumstance?

Exercise 2: Extend the checklist

For a legal document analysis app (high precision required, sensitive data), what 3 specific items would you add to the checklist?

Exercise 3: Automate a new check

Write a function check_guardrails_in_all_routers() for the run_checklist.py script that verifies that each file inside src/app/routers/ contains at least one reference to guardrails or GuardrailsPipeline. It must return (bool, str) indicating whether it passed and the detail.

def check_guardrails_in_all_routers() -> Tuple[bool, str]:
    """Verifies that each router has guardrails configured."""
    from pathlib import Path

    router_dir = Path("src/app/routers")
    if not router_dir.exists():
        return False, "src/app/routers/ directory not found"

    router_files = list(router_dir.glob("*.py"))
    router_files = [f for f in router_files if f.name != "__init__.py"]

    if not router_files:
        return False, "No router files found"

    missing_guardrails = []
    for router_file in router_files:
        content = router_file.read_text()
        if "guardrail" not in content.lower() and "GuardrailsPipeline" not in content:
            missing_guardrails.append(router_file.name)

    if missing_guardrails:
        return False, f"Routers without guardrails: {', '.join(missing_guardrails)}"

    return True, f"All {len(router_files)} routers have guardrails"

Exercise 4: Checklist for a multi-model system

Your application now uses two models: gpt-4o for complex analysis and gpt-4o-mini for fast classification. Write 4 additional checklist items specific to this multi-model configuration, following the checklist format (verify + fix).

MULTI-MODEL 1: BOTH MODELS CONFIGURED
    ├── Verify:
    │   python -c "from src.config import get_settings; s=get_settings(); print(s.openai_model, s.secondary_model)"
    │   → Must return both models, not None
    └── Fix: add secondary_model in .env and in Settings

MULTI-MODEL 2: COST TRACKING PER MODEL
    ├── Verify:
    │   jq 'select(.model) | {model, cost_usd}' logs/app.json | sort | uniq -c
    │   → Each model must have its own cost record
    └── Fix: verify that calculate_cost receives the model name from the provider

MULTI-MODEL 3: FALLBACK BETWEEN MODELS
    ├── Verify:
    │   pytest tests/ -k "test_model_fallback" -v
    │   → If gpt-4o fails, gpt-4o-mini must take the request
    └── Fix: configure FallbackProvider with both models

MULTI-MODEL 4: INDEPENDENT RATE LIMITS
    ├── Verify:
    │   grep -r "RateLimitedProvider" src/ → must have one per model
    │   → Each model has its own rate limit (gpt-4o: 500 RPM, mini: 2000 RPM)
    └── Fix: create a separate RateLimitedProvider per model in dependencies.py

Troubleshooting

Problem: The checklist script fails with ModuleNotFoundError

Symptom: When running python scripts/run_checklist.py you get ModuleNotFoundError: No module named 'src'.

Cause: The script runs from a directory where Python can't find the src package.

Solution:

# Option 1: run from the project root
cd /path/to/project
python scripts/run_checklist.py

# Option 2: add the path at the start of the script
import sys, os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

# Option 3: install the package in editable mode
pip install -e .

Problem: grep -r "sk-" src/ gives false positives

Symptom: The secrets check reports finding sk- but they're variable names like skip, task-id, or comments.

Cause: The sk- pattern is too generic.

Solution:

# Use a more specific pattern for OpenAI API keys
grep -rP "sk-[a-zA-Z0-9]{20,}" src/

# Or exclude common words
grep -r "sk-" src/ | grep -v "skip\|task\|mask\|desk"

# Better yet: use dedicated tools like gitleaks or trufflehog
pip install trufflehog
trufflehog filesystem --directory src/

Problem: The guardrails tests pass locally but fail in CI

Symptom: pytest tests/ -k "guardrail" passes on your machine but fails in GitHub Actions with a timeout or connection errors.

Cause: The guardrails tests make real calls to the LLM on your machine (with your API key) but in CI there's no API key or there's rate limiting.

Solution:

# In conftest.py: use a mock for CI
import pytest
import os

@pytest.fixture
def guardrails_pipeline():
    """Guardrails pipeline that doesn't need a real LLM."""
    return GuardrailsPipeline(
        injection_enabled=True,
        content_policy_enabled=True,
        pii_redaction_enabled=True,
    )

@pytest.mark.skipif(
    not os.environ.get("OPENAI_API_KEY"),
    reason="Requires OPENAI_API_KEY"
)
def test_guardrail_with_real_llm():
    ...

Problem: The health check says 200 but the app doesn't process requests

Symptom: curl /health/live returns 200, but requests to /analyze fail with 500.

Cause: The liveness health check only verifies that the HTTP process is running. It doesn't verify external dependencies like OpenAI.

Solution:

# Use /health/ready instead of /health/live to validate dependencies
# curl http://localhost:8000/health/ready

@router.get("/health/ready")
async def readiness(provider: LLMProvider = Depends(get_llm_provider)):
    try:
        provider.complete([{"role": "user", "content": "ping"}])
        return {"status": "ready"}
    except Exception as e:
        raise HTTPException(status_code=503, detail=str(e))

The key distinction between the three health endpoints:

• /health/live — is the process running? (liveness probe)
• /health/ready — can it accept traffic? (readiness probe, includes dependencies)
• /health/deps — which dependencies are active? (detailed diagnosis)

Summary

• Specific to AI: the checklist covers what generic checklists ignore (versioned prompts, guardrails, cost tracking, fallbacks)
• Executable: each item has a concrete command to verify and a concrete step to fix
• 6 categories: Security, Testing, Observability, Reliability, Performance, Documentation
• Automatable: the repeatable items are coded in scripts/run_checklist.py
• Before each deploy: the checklist is not a one-time thing — it runs before each launch

Additional resources

1. Google SRE Book — Production Readiness Review — Google's framework for validating apps before production
2. 12-Factor App — Configuration and deployment methodology
3. OWASP LLM Top 10 — The 10 most critical vulnerabilities in LLM apps
4. Snyk — Secrets and vulnerability scanning tool