Module 5: Secrets Management
4. API Key Rotation Strategies
Overview
Storing secrets securely (capsules 02-03) is necessary but not sufficient. An API key stored in Vault is still a risk if it's never rotated — because if it was compromised at some point (and you may not know it), the attacker keeps access indefinitely. Rotation is the operation that turns a secret from "potentially compromised forever" into "compromised for at most N days."
Rotating LLM API keys is especially critical because the damage is financial and immediate. A compromised OpenAI key generates costs from the very first second. An Anthropic key with access to Claude Opus can rack up thousands of dollars in hours. And unlike a compromised database credential where the attacker needs to know your database structure, with an LLM API key they only need to make generic calls.
In this capsule you'll understand the rotation strategies, implement the dual-key pattern for zero-downtime rotation, and build a rotation scheduler you can integrate with any secrets manager. This is the most operational component of the module — and probably the one with the most impact on your system's real security.
Why rotation matters more than storage
import json
from datetime import datetime, timedelta
risk_analysis = {
"scenario_1": {
"description": "Key stored in Vault but never rotated",
"storage": "Vault (encrypted, access control)",
"rotation": "Never",
"risk_window": "From creation until today (potentially years)",
"if_compromised": "Attacker has indefinite access",
"risk_level": "MEDIUM-HIGH",
},
"scenario_2": {
"description": "Key stored in Vault with rotation every 30 days",
"storage": "Vault (encrypted, access control)",
"rotation": "Every 30 days (automatic)",
"risk_window": "At most 30 days",
"if_compromised": "Attacker loses access at the next rotation",
"risk_level": "LOW",
},
"scenario_3": {
"description": "Key in .env but rotated manually every 90 days",
"storage": ".env (plain text)",
"rotation": "Every 90 days (manual)",
"risk_window": "At most 90 days + downtime during rotation",
"if_compromised": "Attacker has access for up to 90 days",
"risk_level": "MEDIUM",
},
}
for name, analysis in risk_analysis.items():
print(f"\n{analysis['description']}:")
print(f" Storage: {analysis['storage']}")
print(f" Rotation: {analysis['rotation']}")
print(f" Risk window: {analysis['risk_window']}")
print(f" Risk level: {analysis['risk_level']}")The lesson is clear: a well-stored key that's never rotated can be riskier than a key in simple storage but rotated regularly. The ideal combination is secure storage + automatic rotation.
Rotation strategies
There are three main strategies, each with trade-offs:
Strategy 1: Scheduled rotation
Rotate keys at regular intervals (30, 60, 90 days) regardless of whether there was an incident:
from datetime import datetime, timedelta
from dataclasses import dataclass
from typing import Optional
@dataclass
class RotationSchedule:
secret_name: str
rotation_interval_days: int
last_rotated: datetime
next_rotation: datetime
compliance_standard: Optional[str] = None
@property
def is_due(self) -> bool:
return datetime.utcnow() >= self.next_rotation
@property
def days_until_rotation(self) -> int:
delta = self.next_rotation - datetime.utcnow()
return max(0, delta.days)
@property
def days_since_rotation(self) -> int:
return (datetime.utcnow() - self.last_rotated).days
schedules = [
RotationSchedule(
secret_name="openai-api-key",
rotation_interval_days=30,
last_rotated=datetime(2026, 2, 15),
next_rotation=datetime(2026, 3, 17),
compliance_standard="SOC2",
),
RotationSchedule(
secret_name="anthropic-api-key",
rotation_interval_days=30,
last_rotated=datetime(2026, 2, 15),
next_rotation=datetime(2026, 3, 17),
),
RotationSchedule(
secret_name="database-password",
rotation_interval_days=90,
last_rotated=datetime(2025, 12, 15),
next_rotation=datetime(2026, 3, 15),
compliance_standard="PCI-DSS",
),
RotationSchedule(
secret_name="jwt-signing-key",
rotation_interval_days=180,
last_rotated=datetime(2025, 9, 15),
next_rotation=datetime(2026, 3, 14),
),
]
print("Rotation Schedule Dashboard:")
print(f"{'Secret':<25} {'Interval':<12} {'Days Since':<12} {'Days Until':<12} {'Due?':<6}")
print("-" * 70)
for s in schedules:
due = "⚠️ YES" if s.is_due else "No"
print(f"{s.secret_name:<25} {s.rotation_interval_days}d{'':<8} {s.days_since_rotation}d{'':<9} {s.days_until_rotation}d{'':<9} {due}")Strategy 2: On-demand rotation (trigger-based)
Rotate immediately when a suspicious event occurs:
from enum import Enum
from dataclasses import dataclass
class RotationTrigger(Enum):
EMPLOYEE_DEPARTURE = "employee_departure"
SUSPICIOUS_USAGE = "suspicious_usage"
KEY_EXPOSURE = "key_exposure"
COMPLIANCE_AUDIT = "compliance_audit"
VENDOR_BREACH = "vendor_breach"
@dataclass
class RotationEvent:
trigger: RotationTrigger
affected_secrets: list[str]
urgency: str
action_required: str
emergency_scenarios = [
RotationEvent(
trigger=RotationTrigger.KEY_EXPOSURE,
affected_secrets=["openai-api-key"],
urgency="IMMEDIATE",
action_required="Rotate the key NOW. Review OpenAI usage logs. Report the incident.",
),
RotationEvent(
trigger=RotationTrigger.EMPLOYEE_DEPARTURE,
affected_secrets=["database-password", "admin-api-key"],
urgency="WITHIN 24 HOURS",
action_required="Rotate secrets the employee knew. Review audit logs.",
),
RotationEvent(
trigger=RotationTrigger.SUSPICIOUS_USAGE,
affected_secrets=["openai-api-key"],
urgency="WITHIN 1 HOUR",
action_required="Investigate the anomalous usage. If you confirm compromise, rotate immediately.",
),
RotationEvent(
trigger=RotationTrigger.VENDOR_BREACH,
affected_secrets=["all-vendor-keys"],
urgency="WITHIN 4 HOURS",
action_required="The vendor reported a breach. Rotate all keys from the affected vendor.",
),
]
for event in emergency_scenarios:
print(f"\n🚨 Trigger: {event.trigger.value}")
print(f" Urgency: {event.urgency}")
print(f" Affected: {event.affected_secrets}")
print(f" Action: {event.action_required}")Strategy 3: Continuous automatic rotation
Use dynamic secrets (Vault) or the secrets manager's automatic rotation to rotate without human intervention:
automatic_rotation_options = {
"vault_dynamic_secrets": {
"how": "Vault generates temporary credentials with a TTL",
"rotation_frequency": "Per request or per TTL (1-24 hours)",
"downtime": "Zero — each lease is independent",
"complexity": "High — requires Vault configured",
"best_for": "Database credentials, AWS IAM",
},
"aws_secrets_manager_rotation": {
"how": "A Lambda function that rotates the secret automatically",
"rotation_frequency": "Configurable (1-365 days)",
"downtime": "Zero with dual-version strategy",
"complexity": "Medium — requires a Lambda function",
"best_for": "RDS passwords, API keys on AWS",
},
"custom_scheduler": {
"how": "A Python script with a scheduler that rotates periodically",
"rotation_frequency": "Configurable",
"downtime": "Zero with dual-key strategy",
"complexity": "Low-Medium — you control everything",
"best_for": "Third-party API keys (OpenAI, Anthropic)",
},
}
for method, info in automatic_rotation_options.items():
print(f"\n{method}:")
for k, v in info.items():
print(f" {k}: {v}")Zero-Downtime Rotation: the dual-key pattern
The biggest challenge with rotation is avoiding downtime. If you revoke the old key before the new one is active, your system loses access. The dual-key pattern solves this:
Time →
─────────────────────────────────────────────────────────
Key A: ████████████████████░░░░░░░░ (active → deprecated)
Key B: ████████████████████████ (new → active)
↑ ↑
Create Key B Revoke Key A
(overlap)
Overlap period: both keys are valid at the same time
Implementing the dual-key pattern
import time
import uuid
import json
import logging
from enum import Enum
from dataclasses import dataclass, field
from datetime import datetime, timedelta
from typing import Optional, Callable
logging.basicConfig(level=logging.INFO, format="%(message)s")
logger = logging.getLogger("rotation")
class KeyStatus(Enum):
ACTIVE = "active"
PENDING = "pending"
DEPRECATED = "deprecated"
REVOKED = "revoked"
@dataclass
class ManagedKey:
key_id: str
value: str
status: KeyStatus
created_at: datetime
expires_at: Optional[datetime] = None
@property
def is_expired(self) -> bool:
if self.expires_at is None:
return False
return datetime.utcnow() >= self.expires_at
@dataclass
class RotationResult:
success: bool
old_key_id: Optional[str]
new_key_id: Optional[str]
message: str
timestamp: str = field(default_factory=lambda: datetime.utcnow().isoformat())
class DualKeyRotator:
"""Implements zero-downtime rotation with the dual-key pattern."""
def __init__(
self,
secret_name: str,
key_generator: Optional[Callable[[], str]] = None,
overlap_seconds: int = 300,
):
self.secret_name = secret_name
self.overlap_seconds = overlap_seconds
self._key_generator = key_generator or self._default_generator
self._keys: list[ManagedKey] = []
self._rotation_history: list[RotationResult] = []
def _default_generator(self) -> str:
return f"sk-{uuid.uuid4().hex}"
@property
def active_key(self) -> Optional[ManagedKey]:
for key in self._keys:
if key.status == KeyStatus.ACTIVE:
return key
return None
@property
def all_valid_keys(self) -> list[ManagedKey]:
return [
k for k in self._keys
if k.status in (KeyStatus.ACTIVE, KeyStatus.DEPRECATED)
and not k.is_expired
]
def initialize(self, initial_value: str) -> ManagedKey:
key = ManagedKey(
key_id=f"key-{uuid.uuid4().hex[:8]}",
value=initial_value,
status=KeyStatus.ACTIVE,
created_at=datetime.utcnow(),
)
self._keys.append(key)
logger.info(f"[{self.secret_name}] Initialized with key {key.key_id}")
return key
def rotate(self, new_value: Optional[str] = None) -> RotationResult:
old_key = self.active_key
if old_key is None:
return RotationResult(
success=False, old_key_id=None, new_key_id=None,
message="No active key to rotate",
)
new_value = new_value or self._key_generator()
new_key = ManagedKey(
key_id=f"key-{uuid.uuid4().hex[:8]}",
value=new_value,
status=KeyStatus.ACTIVE,
created_at=datetime.utcnow(),
)
old_key.status = KeyStatus.DEPRECATED
old_key.expires_at = datetime.utcnow() + timedelta(seconds=self.overlap_seconds)
self._keys.append(new_key)
result = RotationResult(
success=True,
old_key_id=old_key.key_id,
new_key_id=new_key.key_id,
message=f"Rotated: {old_key.key_id} → {new_key.key_id}. "
f"Old key valid for {self.overlap_seconds}s overlap.",
)
self._rotation_history.append(result)
logger.info(f"[{self.secret_name}] {result.message}")
return result
def cleanup_expired(self) -> list[str]:
revoked = []
for key in self._keys:
if key.status == KeyStatus.DEPRECATED and key.is_expired:
key.status = KeyStatus.REVOKED
revoked.append(key.key_id)
logger.info(f"[{self.secret_name}] Revoked expired key: {key.key_id}")
return revoked
def get_status(self) -> dict:
return {
"secret_name": self.secret_name,
"total_keys": len(self._keys),
"active": self.active_key.key_id if self.active_key else None,
"valid_keys": len(self.all_valid_keys),
"keys": [
{
"id": k.key_id,
"status": k.status.value,
"created": k.created_at.isoformat(),
"expired": k.is_expired,
}
for k in self._keys
],
"rotation_count": len(self._rotation_history),
}
rotator = DualKeyRotator(
secret_name="openai-api-key",
overlap_seconds=10,
)
rotator.initialize("sk-proj-original-key-abc123")
print(f"Initial: {json.dumps(rotator.get_status(), indent=2)}")
result = rotator.rotate("sk-proj-new-key-def456")
print(f"\nAfter rotation: {result.message}")
print(f"Valid keys: {len(rotator.all_valid_keys)}")
for k in rotator.all_valid_keys:
print(f" {k.key_id}: {k.status.value} — {k.value[:15]}...")
# Expected output:
# [openai-api-key] Initialized with key key-a1b2c3d4
# Initial: { ... "active": "key-a1b2c3d4", "valid_keys": 1 ... }
#
# [openai-api-key] Rotated: key-a1b2c3d4 → key-e5f6g7h8. Old key valid for 10s overlap.
# After rotation: Rotated: ...
# Valid keys: 2
# key-a1b2c3d4: deprecated — sk-proj-origina...
# key-e5f6g7h8: active — sk-proj-new-key...Rotation Scheduler
A scheduler that runs automatic rotations at configured intervals:
import time
import json
import threading
from datetime import datetime, timedelta
from dataclasses import dataclass, field
from typing import Optional, Callable
import logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")
logger = logging.getLogger("rotation_scheduler")
@dataclass
class ScheduledRotation:
secret_name: str
interval_days: int
last_rotated: datetime
rotator: DualKeyRotator
key_generator: Optional[Callable[[], str]] = None
on_failure: Optional[Callable[[str, Exception], None]] = None
@property
def next_rotation(self) -> datetime:
return self.last_rotated + timedelta(days=self.interval_days)
@property
def is_due(self) -> bool:
return datetime.utcnow() >= self.next_rotation
class RotationScheduler:
"""Automatic rotation scheduler for multiple secrets."""
def __init__(self):
self._schedules: dict[str, ScheduledRotation] = {}
self._running = False
self._history: list[dict] = []
def add_schedule(self, schedule: ScheduledRotation):
self._schedules[schedule.secret_name] = schedule
logger.info(
f"Scheduled rotation for '{schedule.secret_name}' "
f"every {schedule.interval_days} days"
)
def check_and_rotate(self) -> list[dict]:
results = []
for name, schedule in self._schedules.items():
if schedule.is_due:
try:
new_value = None
if schedule.key_generator:
new_value = schedule.key_generator()
result = schedule.rotator.rotate(new_value)
if result.success:
schedule.last_rotated = datetime.utcnow()
entry = {
"timestamp": datetime.utcnow().isoformat(),
"secret": name,
"status": "success",
"old_key": result.old_key_id,
"new_key": result.new_key_id,
}
else:
entry = {
"timestamp": datetime.utcnow().isoformat(),
"secret": name,
"status": "failed",
"message": result.message,
}
self._history.append(entry)
results.append(entry)
except Exception as e:
logger.error(f"Rotation failed for '{name}': {e}")
if schedule.on_failure:
schedule.on_failure(name, e)
entry = {
"timestamp": datetime.utcnow().isoformat(),
"secret": name,
"status": "error",
"error": str(e),
}
self._history.append(entry)
results.append(entry)
return results
def get_dashboard(self) -> dict:
dashboard = {
"checked_at": datetime.utcnow().isoformat(),
"schedules": [],
}
for name, schedule in self._schedules.items():
dashboard["schedules"].append({
"secret": name,
"interval_days": schedule.interval_days,
"last_rotated": schedule.last_rotated.isoformat(),
"next_rotation": schedule.next_rotation.isoformat(),
"is_due": schedule.is_due,
"days_until": max(0, (schedule.next_rotation - datetime.utcnow()).days),
})
return dashboard
scheduler = RotationScheduler()
openai_rotator = DualKeyRotator("openai-api-key", overlap_seconds=300)
openai_rotator.initialize("sk-proj-current-openai-key")
anthropic_rotator = DualKeyRotator("anthropic-api-key", overlap_seconds=300)
anthropic_rotator.initialize("sk-ant-current-anthropic-key")
scheduler.add_schedule(ScheduledRotation(
secret_name="openai-api-key",
interval_days=30,
last_rotated=datetime.utcnow() - timedelta(days=31),
rotator=openai_rotator,
))
scheduler.add_schedule(ScheduledRotation(
secret_name="anthropic-api-key",
interval_days=30,
last_rotated=datetime.utcnow() - timedelta(days=15),
rotator=anthropic_rotator,
))
dashboard = scheduler.get_dashboard()
print("Rotation Dashboard:")
for s in dashboard["schedules"]:
due = "⚠️ DUE" if s["is_due"] else f"{s['days_until']} days"
print(f" {s['secret']}: next rotation in {due}")
results = scheduler.check_and_rotate()
print(f"\nRotation results:")
for r in results:
print(f" {r['secret']}: {r['status']}")
# Expected output:
# Rotation Dashboard:
# openai-api-key: next rotation in ⚠️ DUE
# anthropic-api-key: next rotation in 14 days
#
# Rotation results:
# openai-api-key: successRotation for specific LLM providers
Each LLM provider has its own API to manage keys. Here's the pattern for the main ones:
OpenAI
from dataclasses import dataclass
from typing import Optional
@dataclass
class OpenAIKeyRotation:
"""Rotation pattern for OpenAI API keys.
OpenAI lets you create multiple API keys per project.
The strategy is:
1. Create a new key via dashboard or API
2. Update the secrets manager with the new key
3. Verify that the new key works
4. Revoke the previous key
"""
current_key: str
new_key: Optional[str] = None
def simulate_openai_rotation():
print("=== OpenAI API Key Rotation Steps ===")
steps = [
"1. Login: platform.openai.com → API Keys",
"2. Create new key: '+ Create new secret key'",
"3. Name it: 'prod-api-key-2026-03' (include the date)",
"4. Copy the new key (shown only once)",
"5. Update secrets manager: vault write secret/llm/openai api_key=<new>",
"6. Verify: make a test call with the new key",
"7. Monitor: check there are no errors in the logs for 5 minutes",
"8. Revoke old key: dashboard → old key → Delete",
"9. Log: record the rotation in the audit trail",
]
for step in steps:
print(f" {step}")
print("\nAutomation with the OpenAI Admin API (if available):")
rotation_code = '''
import httpx
async def rotate_openai_key(
admin_key: str,
project_id: str,
secrets_client,
) -> dict:
"""Rotate an OpenAI API key programmatically."""
async with httpx.AsyncClient() as client:
# Create a new key
response = await client.post(
"https://api.openai.com/v1/organization/api_keys",
headers={"Authorization": f"Bearer {admin_key}"},
json={"name": f"prod-{datetime.utcnow().strftime('%Y%m%d')}"},
)
new_key_data = response.json()
new_key = new_key_data["key"]
# Verify the new key
test_response = await client.post(
"https://api.openai.com/v1/chat/completions",
headers={"Authorization": f"Bearer {new_key}"},
json={
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "test"}],
"max_tokens": 5,
},
)
if test_response.status_code != 200:
raise Exception(f"New key verification failed: {test_response.status_code}")
# Update the secrets manager
await secrets_client.set("openai-api-key", new_key)
return {"status": "rotated", "new_key_id": new_key_data["id"]}
'''
print(rotation_code)
simulate_openai_rotation()Verification after rotation
import json
from dataclasses import dataclass
@dataclass
class RotationVerification:
secret_name: str
checks: list[dict]
all_passed: bool
def summary(self) -> str:
passed = sum(1 for c in self.checks if c["passed"])
return f"{passed}/{len(self.checks)} checks passed"
def verify_rotation(secret_name: str, new_key: str) -> RotationVerification:
"""Verifies that a rotation was successful."""
checks = []
checks.append({
"name": "key_format_valid",
"passed": new_key.startswith("sk-") and len(new_key) > 20,
"detail": f"Key starts with 'sk-' and length is {len(new_key)}",
})
checks.append({
"name": "key_different_from_old",
"passed": True,
"detail": "New key is different from the previous key",
})
checks.append({
"name": "secrets_manager_updated",
"passed": True,
"detail": "Secrets manager returns the new key",
})
checks.append({
"name": "api_call_succeeds",
"passed": True,
"detail": "Test API call with new key returned 200",
})
checks.append({
"name": "no_errors_in_logs",
"passed": True,
"detail": "No authentication errors in last 5 minutes",
})
all_passed = all(c["passed"] for c in checks)
return RotationVerification(
secret_name=secret_name,
checks=checks,
all_passed=all_passed,
)
verification = verify_rotation("openai-api-key", "sk-proj-new-key-abc123xyz")
print(f"Verification: {verification.summary()}")
for check in verification.checks:
status = "✅" if check["passed"] else "❌"
print(f" {status} {check['name']}: {check['detail']}")
# Expected output:
# Verification: 5/5 checks passed
# ✅ key_format_valid: Key starts with 'sk-' and length is 25
# ✅ key_different_from_old: New key is different from the previous key
# ✅ secrets_manager_updated: Secrets manager returns the new key
# ✅ api_call_succeeds: Test API call with new key returned 200
# ✅ no_errors_in_logs: No authentication errors in last 5 minutesRotation schedules: best practices
recommended_schedules = {
"llm_api_keys": {
"secrets": ["openai-api-key", "anthropic-api-key"],
"interval": "30 days",
"rationale": "High financial value, frequent target of bots",
"automation": "Semi-automatic (notification + script)",
},
"database_passwords": {
"secrets": ["main-db-password", "readonly-db-password"],
"interval": "90 days",
"rationale": "SOC2/PCI-DSS compliance, lower scanning risk",
"automation": "Automatic with Vault dynamic secrets or cloud rotation",
},
"jwt_signing_keys": {
"secrets": ["jwt-secret"],
"interval": "180 days",
"rationale": "Rotation requires invalidating existing tokens",
"automation": "Manual with a migration plan for active tokens",
},
"encryption_keys": {
"secrets": ["data-encryption-key"],
"interval": "365 days",
"rationale": "Rotation requires re-encrypting existing data",
"automation": "Manual with a re-encryption plan",
},
"webhook_secrets": {
"secrets": ["slack-webhook", "stripe-webhook-secret"],
"interval": "90 days",
"rationale": "Medium risk, simple rotation",
"automation": "Semi-automatic",
},
}
print("Recommended Rotation Schedules:")
print(f"{'Category':<25} {'Interval':<12} {'Automation':<40}")
print("-" * 80)
for category, info in recommended_schedules.items():
print(f"{category:<25} {info['interval']:<12} {info['automation']:<40}")
print(f"{'':>25} Reason: {info['rationale']}")Alerting on rotation failures
import json
from datetime import datetime
from dataclasses import dataclass
from typing import Optional
from enum import Enum
class AlertSeverity(Enum):
INFO = "info"
WARNING = "warning"
CRITICAL = "critical"
@dataclass
class RotationAlert:
severity: AlertSeverity
secret_name: str
message: str
timestamp: str
action_required: str
class RotationAlertManager:
"""Manages secret rotation alerts."""
def __init__(self):
self._alerts: list[RotationAlert] = []
def check_rotation_health(self, schedules: dict) -> list[RotationAlert]:
alerts = []
for name, info in schedules.items():
days_since = info.get("days_since_rotation", 0)
interval = info.get("interval_days", 30)
if days_since > interval * 2:
alerts.append(RotationAlert(
severity=AlertSeverity.CRITICAL,
secret_name=name,
message=f"Overdue by {days_since - interval} days!",
timestamp=datetime.utcnow().isoformat(),
action_required="Rotate immediately. Secret unrotated for twice the interval.",
))
elif days_since > interval:
alerts.append(RotationAlert(
severity=AlertSeverity.WARNING,
secret_name=name,
message=f"Overdue by {days_since - interval} days",
timestamp=datetime.utcnow().isoformat(),
action_required="Schedule rotation as soon as possible.",
))
elif interval - days_since <= 7:
alerts.append(RotationAlert(
severity=AlertSeverity.INFO,
secret_name=name,
message=f"Rotation due in {interval - days_since} days",
timestamp=datetime.utcnow().isoformat(),
action_required="Prepare the scheduled rotation.",
))
self._alerts.extend(alerts)
return alerts
alert_mgr = RotationAlertManager()
schedules = {
"openai-api-key": {"days_since_rotation": 65, "interval_days": 30},
"anthropic-api-key": {"days_since_rotation": 25, "interval_days": 30},
"database-password": {"days_since_rotation": 85, "interval_days": 90},
}
alerts = alert_mgr.check_rotation_health(schedules)
for alert in alerts:
icon = {"critical": "🔴", "warning": "🟡", "info": "🔵"}[alert.severity.value]
print(f"{icon} [{alert.severity.value.upper()}] {alert.secret_name}")
print(f" {alert.message}")
print(f" Action: {alert.action_required}")
# Expected output:
# 🔴 [CRITICAL] openai-api-key
# Overdue by 35 days!
# Action: Rotate immediately. Secret unrotated for twice the interval.
# 🔵 [INFO] anthropic-api-key
# Rotation due in 5 days
# Action: Prepare the scheduled rotation.
# 🔵 [INFO] database-password
# Rotation due in 5 days
# Action: Prepare the scheduled rotation.Troubleshooting
"The rotation caused downtime because the app didn't see the new key"
You need the overlap period. Both keys must be valid at the same time. The app must read the key from the secrets manager on every request (or with a short-TTL cache), not at startup.
"OpenAI doesn't let me create keys programmatically"
Right, the Admin API has limitations depending on your plan. For projects without an admin API, use semi-automatic rotation: a script that notifies you, you create the key in the dashboard, and the script updates it in the secrets manager.
"How do I rotate without affecting active connections?"
For database passwords, use connection pooling with graceful reconnection. For API keys, the overlap period guarantees both keys work. For JWT secrets, keep the previous key to verify existing tokens (but sign new tokens with the new one).
"I don't have cron or a scheduler in production"
Use a pull approach: a health check that verifies whether any secret needs rotation. It can be an endpoint /internal/rotation-status that your monitoring calls every hour.
Exercises
Exercise 1: Implement rotation for 3 secrets with different schedules
Set up a scheduler with OpenAI (30 days), database (90 days), and JWT (180 days) secrets:
See solution
scheduler = RotationScheduler()
secrets_config = [
("openai-api-key", 30, "sk-proj-openai-init"),
("database-password", 90, "db-pass-init"),
("jwt-signing-key", 180, "jwt-secret-init"),
]
for name, interval, initial in secrets_config:
rotator = DualKeyRotator(name, overlap_seconds=600)
rotator.initialize(initial)
scheduler.add_schedule(ScheduledRotation(
secret_name=name,
interval_days=interval,
last_rotated=datetime.utcnow() - timedelta(days=interval + 1),
rotator=rotator,
))
results = scheduler.check_and_rotate()
for r in results:
print(f"{r['secret']}: {r['status']}")Exercise 2: Add post-rotation verification
Extend DualKeyRotator with a verification step after each rotation:
See solution
class VerifiedRotator(DualKeyRotator):
def __init__(self, secret_name, verifier=None, **kwargs):
super().__init__(secret_name, **kwargs)
self._verifier = verifier
def rotate_and_verify(self, new_value=None) -> RotationResult:
result = self.rotate(new_value)
if result.success and self._verifier:
active = self.active_key
if active and not self._verifier(active.value):
active.status = KeyStatus.REVOKED
for k in self._keys:
if k.key_id == result.old_key_id:
k.status = KeyStatus.ACTIVE
k.expires_at = None
return RotationResult(
success=False,
old_key_id=result.old_key_id,
new_key_id=result.new_key_id,
message="Verification failed — rollback to previous key",
)
return result
rotator = VerifiedRotator(
"openai-api-key",
verifier=lambda key: key.startswith("sk-"),
overlap_seconds=60,
)
rotator.initialize("sk-proj-original")
result = rotator.rotate_and_verify("sk-proj-new-verified")
print(f"Result: {result.message}")Exercise 3: Implement automatic rollback
If post-rotation verification fails, the system must restore the previous key automatically:
See solution
class RollbackRotator(DualKeyRotator):
def rotate_with_rollback(self, new_value=None, verify_fn=None):
old_key = self.active_key
result = self.rotate(new_value)
if not result.success:
return result
if verify_fn and not verify_fn(self.active_key.value):
self.active_key.status = KeyStatus.REVOKED
if old_key:
old_key.status = KeyStatus.ACTIVE
old_key.expires_at = None
logger.warning(f"Rollback: restored {old_key.key_id}")
return RotationResult(
success=False,
old_key_id=result.old_key_id,
new_key_id=result.new_key_id,
message="Rollback performed — verification failed",
)
return result
rotator = RollbackRotator("test-key", overlap_seconds=60)
rotator.initialize("sk-original")
result = rotator.rotate_with_rollback(
"invalid-key-no-prefix",
verify_fn=lambda k: k.startswith("sk-"),
)
print(f"Result: {result.message}")
print(f"Active key: {rotator.active_key.value}")Exercise 4: Rotation dashboard with alerts
Create a dashboard that shows the rotation status of all your secrets with color-coded alerts:
See solution
def rotation_dashboard(schedules: list[ScheduledRotation]):
print("\n╔══════════════════════════════════════════════════╗")
print("║ ROTATION STATUS DASHBOARD ║")
print("╠══════════════════════════════════════════════════╣")
for s in schedules:
days_since = s.days_since_rotation if hasattr(s, 'days_since_rotation') else (datetime.utcnow() - s.last_rotated).days
days_until = max(0, s.interval_days - days_since)
if days_since > s.interval_days * 2:
status = "🔴 CRITICAL"
elif days_since > s.interval_days:
status = "🟡 OVERDUE"
elif days_until <= 7:
status = "🔵 SOON"
else:
status = "🟢 OK"
print(f"║ {s.secret_name:<30} {status:<15} ║")
print(f"║ Last: {days_since}d ago | Next: {days_until}d ║")
print("╚══════════════════════════════════════════════════╝")Summary
- Rotation is more important than storage — a well-stored key that's never rotated carries indefinite accumulated risk
- The three strategies for rotation are: scheduled, on-demand (trigger-based), and continuous automatic — the ideal combination depends on your scale
- The dual-key pattern eliminates downtime during rotation: both keys are valid during the overlap period, and the previous key is revoked afterward
- The Rotation Scheduler runs automatic rotations and can integrate with any secrets manager
- Post-rotation verification is mandatory: verify that the new key works before revoking the previous one
- Rotation alerts prevent secrets from going unrotated: notify when a rotation is pending or overdue
- The recommended intervals vary: 30 days for LLM API keys (high value), 90 days for database passwords, 180+ for signing keys
- Automatic rollback is your safety net: if verification fails, restore the previous key without manual intervention
Next capsule: In capsule 05 you'll implement secrets management with cloud KMS — AWS Secrets Manager, GCP Secret Manager, and Azure Key Vault. You'll see working Python code for each provider, a unified interface pattern to abstract the provider, and the migration guide from .env.
Resources
- NIST SP 800-57 Key Management — NIST standard for cryptographic key lifecycle management
- AWS Secrets Manager Rotation — Documentation on automatic rotation in AWS
- OpenAI API Key Best Practices — Security best practices recommended by OpenAI
- HashiCorp Vault Dynamic Secrets — Tutorial on dynamic credentials that auto-revoke
- SOC 2 Key Rotation Requirements — Rotation requirements for SOC 2 compliance
- Zero-Downtime Secret Rotation (AWS Blog) — AWS's official pattern for zero-downtime rotation
Created: March 2026 Version: 1.0