6. Environment Management

Description

An AI app has radically different behavior in dev, staging, and production. In dev, you want mock LLM and debug logging to iterate fast. In staging, you want the real LLM but with budget caps to avoid overspending. In production, you want maximum security, full guardrails, and observability. This capsule implements an environment management system that makes that differentiation automatically.

The 3 environments and their differences

┌────────────────────────────────────────────────────────────────────────┐
│                         ENVIRONMENT COMPARISON                         │
├──────────────────┬──────────────────┬──────────────────┬───────────────┤
│ Feature          │ Development      │ Staging          │ Production    │
├──────────────────┼──────────────────┼──────────────────┼───────────────┤
│ LLM Provider     │ Mock (no API)    │ Real (OpenAI)    │ Real (OpenAI) │
│ API Key          │ Not required     │ Sandbox key      │ Prod key      │
│ Log Level        │ DEBUG            │ INFO             │ INFO          │
│ Full Prompt Log  │ Yes              │ No               │ No            │
│ Guardrails       │ Disabled OK      │ Full             │ Full          │
│ PII Redaction    │ Optional         │ Active           │ Active        │
│ Budget/day       │ $0 (mock)        │ $5-10            │ $50-500       │
│ Retry attempts   │ 1 (fails fast)   │ 3                │ 3             │
│ Mock LLM         │ ✅ allowed       │ ❌ not allowed   │ ❌ forbidden  │
│ Debug flag       │ true             │ false            │ false         │
│ .env in repo     │ Partially        │ Only non-secrets │ Never         │
│ Fail on startup  │ No               │ No               │ Yes (strict)  │
└──────────────────┴──────────────────┴──────────────────┴───────────────┘

Settings with per-environment behavior

# src/config.py
from typing import Literal
from pydantic import Field, SecretStr, model_validator, field_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
from functools import lru_cache
import os
from pathlib import Path

def _env_files() -> list[str]:
    """Determine which .env files to load."""
    env = os.getenv("ENVIRONMENT", "development")
    files = []
    
    # Common base (lower priority)
    if Path(".env").exists():
        files.append(".env")
    
    # Per-environment override (higher priority)
    env_specific = f".env.{env}"
    if Path(env_specific).exists():
        files.append(env_specific)
    
    # Personal local override (highest priority among files)
    if Path(".env.local").exists():
        files.append(".env.local")
    
    return files

class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file=_env_files(),
        env_file_encoding="utf-8",
        extra="ignore"
    )
    
    # ─── Environment ──────────────────────────────────────────────
    environment: Literal["development", "staging", "production", "testing"] = "development"
    
    # ─── LLM ─────────────────────────────────────────────────────
    openai_api_key: SecretStr = Field(default=SecretStr(""))
    model: str = "gpt-4o-mini"
    temperature: float = 0.0
    max_tokens: int = 500
    seed: int = None
    use_mock_llm: bool = False
    mock_response: str = '{"sentiment": "positive", "score": 0.8, "confidence": 0.9}'
    
    # ─── Retry ────────────────────────────────────────────────────
    retry_attempts: int = 3
    retry_delay_seconds: float = 1.0
    
    # ─── Logging ─────────────────────────────────────────────────
    log_level: str = "INFO"
    log_full_prompt: bool = False   # Only in debug
    log_to_file: bool = False
    log_file_path: str = "logs/app.json"
    
    # ─── Guardrails ───────────────────────────────────────────────
    enable_guardrails: bool = True
    enable_pii_redaction: bool = True
    enable_injection_check: bool = True
    enable_content_filter: bool = True
    max_input_chars: int = 10_000
    
    # ─── Budget ───────────────────────────────────────────────────
    max_cost_per_request_usd: float = 0.10
    daily_budget_usd: float = 10.0
    
    # ─── Feature flags ────────────────────────────────────────────
    debug: bool = False
    
    # ─── Validators ──────────────────────────────────────────────
    
    @model_validator(mode="after")
    def apply_environment_defaults(self) -> "Settings":
        """
        Applies environment-specific default configurations.
        Called AFTER all fields have been set.
        
        This allows explicit configuration (env vars) to take
        priority over the environment defaults.
        """
        if self.environment == "development":
            # In dev: optimize for iteration speed
            # (Only if not set explicitly)
            if not self.use_mock_llm:
                # Don't force mock, but log it if the dev configured it explicitly
                pass
        
        return self
    
    @model_validator(mode="after")
    def validate_environment_constraints(self) -> "Settings":
        """Environment-specific validations — some are hard prohibitions."""
        
        if self.environment == "production":
            # In production, these are hard constraints — they can't be overridden
            if self.use_mock_llm:
                raise ValueError(
                    "PRODUCTION: use_mock_llm is not allowed. "
                    "Set USE_MOCK_LLM=false"
                )
            
            if not self.openai_api_key.get_secret_value():
                raise ValueError(
                    "PRODUCTION: openai_api_key is required. "
                    "Set OPENAI_API_KEY=sk-..."
                )
            
            if self.log_level == "DEBUG":
                raise ValueError(
                    "PRODUCTION: log_level=DEBUG can expose sensitive information. "
                    "Use INFO or WARNING"
                )
            
            if self.log_full_prompt:
                raise ValueError(
                    "PRODUCTION: log_full_prompt=True can expose PII. "
                    "Set LOG_FULL_PROMPT=false"
                )
        
        if self.environment == "staging":
            if self.use_mock_llm:
                raise ValueError(
                    "STAGING: use_mock_llm is not allowed. "
                    "The goal of staging is to test with the real LLM."
                )
        
        return self
    
    # ─── Properties ───────────────────────────────────────────────
    
    @property
    def is_production(self) -> bool:
        return self.environment == "production"
    
    @property
    def is_development(self) -> bool:
        return self.environment in ("development", "testing")
    
    @property
    def effective_log_level(self) -> str:
        """Effective log level based on environment and debug flag."""
        if self.debug and self.environment != "production":
            return "DEBUG"
        return self.log_level
    
    def create_openai_client(self):
        from openai import OpenAI
        key = self.openai_api_key.get_secret_value()
        if not key:
            raise ValueError("OPENAI_API_KEY not configured")
        return OpenAI(api_key=key)

@lru_cache()
def get_settings() -> Settings:
    return Settings()

.env files per environment

# .env (base — committed, no secrets)
APP_NAME=sentiment-analyzer
APP_VERSION=0.1.0
MODEL=gpt-4o-mini
TEMPERATURE=0.0
MAX_TOKENS=500
RETRY_ATTEMPTS=3
LOG_LEVEL=INFO
ENABLE_GUARDRAILS=true
ENABLE_PII_REDACTION=true
ENABLE_INJECTION_CHECK=true
ENABLE_CONTENT_FILTER=true

# .env.development (committed — dev override)
ENVIRONMENT=development
USE_MOCK_LLM=true
LOG_LEVEL=DEBUG
LOG_FULL_PROMPT=true
DEBUG=true
DAILY_BUDGET_USD=1.0
ENABLE_INJECTION_CHECK=false

# .env.staging (committed — staging override)
ENVIRONMENT=staging
USE_MOCK_LLM=false
LOG_LEVEL=INFO
LOG_FULL_PROMPT=false
DEBUG=false
DAILY_BUDGET_USD=10.0
MAX_COST_PER_REQUEST_USD=0.05
# OPENAI_API_KEY → environment variable on the staging server, NOT in this file

# .env.production (in .gitignore OR empty — only documents the expected variables)
ENVIRONMENT=production
USE_MOCK_LLM=false
LOG_LEVEL=INFO
LOG_FULL_PROMPT=false
DEBUG=false
DAILY_BUDGET_USD=50.0
# OPENAI_API_KEY → secret manager (AWS Secrets Manager, Vault, etc.)

# .env.example (committed — documentation of all variables)
# Copy to .env and configure the values

# ENVIRONMENT (required)
ENVIRONMENT=development    # development | staging | production | testing

# API (required in staging and production)
OPENAI_API_KEY=            # sk-...

# LLM (optional, has defaults)
MODEL=gpt-4o-mini
TEMPERATURE=0.0
MAX_TOKENS=500

# LOGGING
LOG_LEVEL=INFO             # DEBUG | INFO | WARNING | ERROR
LOG_FULL_PROMPT=false      # Only in development

# MOCK (only in development/testing)
USE_MOCK_LLM=false

# BUDGET
DAILY_BUDGET_USD=10.0
MAX_COST_PER_REQUEST_USD=0.10

Startup checks: fail fast in production

# src/startup.py
"""
Checks that run at app startup.
In production, a failed check must prevent the app from starting.
"""
import structlog
from src.config import get_settings

log = structlog.get_logger()

def run_startup_checks() -> None:
    """
    Run all startup checks.
    Raises SystemExit if any check fails in production.
    """
    settings = get_settings()
    checks_passed = True
    
    log.info(
        "startup_checks_starting",
        environment=settings.environment,
        model=settings.model,
        use_mock=settings.use_mock_llm
    )
    
    # Check 1: API key (only if not mock)
    if not settings.use_mock_llm:
        if not settings.openai_api_key.get_secret_value():
            log.error("startup_check_failed", check="openai_api_key",
                     reason="OPENAI_API_KEY not configured")
            checks_passed = False
        else:
            log.info("startup_check_passed", check="openai_api_key")
    
    # Check 2: LLM connectivity (only in staging/production)
    if settings.environment in ("staging", "production") and not settings.use_mock_llm:
        try:
            client = settings.create_openai_client()
            # Minimal test call
            models = client.models.list()
            log.info("startup_check_passed", check="openai_connectivity",
                    models_available=len(list(models)))
        except Exception as e:
            log.error("startup_check_failed", check="openai_connectivity",
                     error=str(e)[:100])
            if settings.is_production:
                checks_passed = False
            else:
                log.warning("startup_check_degraded",
                           check="openai_connectivity",
                           note="Doesn't block staging but report to the team")
    
    # Check 3: Required directories
    if settings.log_to_file:
        import os
        log_dir = os.path.dirname(settings.log_file_path)
        if not os.path.exists(log_dir):
            try:
                os.makedirs(log_dir, exist_ok=True)
                log.info("startup_check_passed", check="log_directory",
                        created=True)
            except OSError as e:
                log.error("startup_check_failed", check="log_directory",
                         error=str(e))
                checks_passed = False
    
    if not checks_passed and settings.is_production:
        log.error("startup_checks_failed", action="exiting")
        raise SystemExit(1)
    
    log.info(
        "startup_checks_completed",
        all_passed=checks_passed,
        environment=settings.environment
    )

# In main.py:
# run_startup_checks()
# app = create_app()

Feature flags per environment

# src/feature_flags.py
"""
Simple feature flags based on configuration.
For more dynamic flags, use LaunchDarkly or another service.
"""
from src.config import get_settings

def is_guardrails_enabled() -> bool:
    settings = get_settings()
    return settings.enable_guardrails

def is_pii_redaction_enabled() -> bool:
    settings = get_settings()
    return settings.enable_pii_redaction

def should_log_full_prompt() -> bool:
    settings = get_settings()
    # Only log the full prompt if it's explicitly enabled
    # and we're not in production
    return settings.log_full_prompt and not settings.is_production

def get_daily_budget_usd() -> float:
    return get_settings().daily_budget_usd

# Usage in the code:
# if is_pii_redaction_enabled():
#     text = redact_pii(text)
#
# if should_log_full_prompt():
#     log.debug("full_prompt", prompt=prompt)

Tests with environment override

# tests/conftest.py
import pytest
from unittest.mock import patch
from src.config import Settings, get_settings

@pytest.fixture
def dev_settings() -> Settings:
    """Development settings for unit tests."""
    return Settings(
        environment="development",
        use_mock_llm=True,
        log_level="WARNING",  # Silence logs in tests
        debug=False,
        openai_api_key="sk-test-key",
        daily_budget_usd=0.0  # No budget in tests
    )

@pytest.fixture
def prod_settings() -> Settings:
    """Production settings to verify the validations."""
    # No real openai_api_key — it's not for calling the LLM
    return Settings(
        environment="production",
        use_mock_llm=False,
        log_level="INFO",
        openai_api_key="sk-prod-test-key",
        daily_budget_usd=50.0
    )

@pytest.fixture(autouse=True)
def override_settings_for_tests(dev_settings):
    """Automatic override for all tests."""
    get_settings.cache_clear()
    with patch("src.config.get_settings", return_value=dev_settings):
        yield dev_settings
    get_settings.cache_clear()


# tests/unit/test_environment_validation.py

def test_production_rejects_mock_llm():
    """In production, use_mock_llm=True must fail validation."""
    with pytest.raises(ValueError, match="PRODUCTION"):
        Settings(
            environment="production",
            use_mock_llm=True,
            openai_api_key="sk-valid-key",
            log_level="INFO"
        )

def test_production_requires_api_key():
    """In production, an empty openai_api_key must fail validation."""
    with pytest.raises(ValueError, match="openai_api_key is required"):
        Settings(
            environment="production",
            use_mock_llm=False,
            openai_api_key="",
            log_level="INFO"
        )

def test_production_rejects_debug_log_level():
    """In production, log_level=DEBUG must fail validation."""
    with pytest.raises(ValueError, match="log_level=DEBUG"):
        Settings(
            environment="production",
            use_mock_llm=False,
            openai_api_key="sk-valid-key",
            log_level="DEBUG"
        )

def test_staging_rejects_mock_llm():
    """In staging, use_mock_llm=True must fail validation."""
    with pytest.raises(ValueError, match="STAGING"):
        Settings(
            environment="staging",
            use_mock_llm=True,
            openai_api_key="sk-valid-key"
        )

def test_development_allows_mock_and_debug():
    """In development, mock and debug are allowed."""
    settings = Settings(
        environment="development",
        use_mock_llm=True,
        log_level="DEBUG",
        debug=True
    )
    assert settings.use_mock_llm is True
    assert settings.log_level == "DEBUG"

Exercises

Exercise 1: Configuration matrix

For your sentiment analysis app, create the complete per-environment configuration table. Include at least 8 variables:

Exercise 2: Add a startup check

Write a startup check that verifies the configured model exists in OpenAI (hint: client.models.retrieve(model)):

def check_model_exists(settings: Settings) -> bool:
    if settings.use_mock_llm:
        return True  # Don't verify in mock
    
    try:
        client = settings.create_openai_client()
        client.models.retrieve(settings.model)
        log.info("startup_check_passed", check="model_exists", model=settings.model)
        return True
    except Exception as e:
        log.error("startup_check_failed", check="model_exists",
                 model=settings.model, error=str(e)[:100])
        return False

Summary

• 3 clear environments: development (mock, debug), staging (real LLM, limited budget), production (full guardrails, monitoring)
• Per-environment validations in pydantic-settings: hard prohibitions that can't be overridden (use_mock_llm in production)
• Startup checks: verify configuration before the first request — "fail fast"
• .env files: .env base + .env.{env} override, secrets only in the server's environment variables
• Feature flags: derived from Settings, simple and predictable

Additional resources

1. 12-Factor App — Dev/Prod Parity — Principle of parity between environments
2. Secret Management (AWS) — For secrets in production
3. HashiCorp Vault — Open source secret manager
4. Feature Flags Best Practices — Martin Fowler
5. Docker Compose for environments — Managing env vars in Docker