6. Secret Rotation

Overview

Rotating secrets means replacing an active API key with a new one. It's not a complex concept, but doing it wrong can break your pipeline, cause downtime, or leave a compromised key active for longer than necessary.

Most teams treat secret rotation as something "we'll do when there's an incident." The problem is that when there is an incident — an exposed key, an employee leaving the company, a provider reporting suspicious access — you need to rotate immediately and without mistakes. If you've never practiced the process, you'll improvise under pressure.

This capsule gives you the complete process, step by step, with edge cases and automation. By the end, you'll be able to rotate an OpenAI API key in your CI/CD pipeline in under 5 minutes without causing downtime.

What you're going to learn:

1. The complete rotation process in 4 steps
2. Edge cases that cause problems during rotation
3. The recommended rotation frequency
4. Partial automation of the process
5. What to do when a key is exposed (emergency rotation)

The rotation process in 4 steps

The process diagram

Step 1: Generate a new key in the provider
    ↓
Step 2: Update the secret on GitHub
    ↓
Step 3: Verify that the pipeline works with the new key
    ↓
Step 4: Revoke the previous key in the provider

Never invert steps 3 and 4. If you revoke the previous key before verifying that the new one works, and the new one has a problem, your pipeline breaks and you have no working key.

Step 1: Generate a new key

OpenAI Dashboard → API Keys → Create new secret key
  Name: "ci-pipeline-2026-Q2" (use a descriptive name with the date)
  Project: your-project
  Permissions: the ones your pipeline needs
  → Copy the new key (sk-proj-new-xyz789...)

The previous key (sk-proj-old-abc123...) stays active. Now you have two valid keys at the same time. This is intentional — you need the overlap window.

Step 2: Update the secret on GitHub

# Update the repository secret
gh secret set OPENAI_API_KEY --body "sk-proj-new-xyz789..."

# Or if you use per-environment secrets
gh secret set OPENAI_API_KEY --env staging --body "sk-proj-new-xyz789..."
gh secret set OPENAI_API_KEY --env production --body "sk-proj-new-xyz789..."

From the UI:

Settings → Secrets → OPENAI_API_KEY → Update secret
  New value: sk-proj-new-xyz789...
  → Update secret

After updating, the workflows that run will use the new key. Workflows that are already running will keep using the previous key (it was injected when the workflow started).

Step 3: Verify that it works

# Trigger a workflow manually to verify
gh workflow run ai-quality-gate.yml

# Wait for it to finish
gh run watch

# Verify the result
gh run list --limit 1

If the workflow passes, the new key works. If it fails, check the logs:

gh run view --log-failed

Common errors at this step:

• 401 Unauthorized: The new key was copied incorrectly
• 429 Rate Limited: The new key has stricter rate limits
• 403 Forbidden: The new key doesn't have the necessary permissions

Step 4: Revoke the previous key

Only after verifying that the new key works:

OpenAI Dashboard → API Keys
  → Find "ci-pipeline-2026-Q1" (the previous key)
  → Revoke
  → Confirm

After revoking, any request with the previous key returns 401. If you had a workflow running with the previous key, that workflow will fail on its next API call — but it's a run in progress, not a systemic problem.

Edge cases

Edge case 1: A workflow running during the rotation

Timeline:
  T0: Workflow A starts with the old key (sk-old...)
  T1: You update the secret on GitHub (sk-new...)
  T2: Workflow A makes an API call with sk-old... → it works (the key is still active)
  T3: You revoke sk-old...
  T4: Workflow A makes another API call with sk-old... → 401 failure

Solution: Wait for the in-progress workflows to finish before revoking the previous key:

# See the active workflows
gh run list --status in_progress

# Wait for them to finish
gh run watch <run-id>

# Now, and only now, revoke the previous key

Edge case 2: Multiple environments with the same key

If staging and production use the same key (not recommended, but it happens):

# Update EVERY environment
gh secret set OPENAI_API_KEY --env staging --body "sk-proj-new..."
gh secret set OPENAI_API_KEY --env production --body "sk-proj-new..."

# Verify BOTH environments
gh workflow run ai-quality-gate.yml  # staging
gh workflow run deploy-production.yml  # production

# Only revoke the previous key when BOTH pass

Edge case 3: The new key doesn't work

Step 1: You generate sk-new...
Step 2: You update the secret
Step 3: The workflow fails with sk-new... → ERROR

What do you do?
  → Do NOT revoke sk-old... (you still need it)
  → Restore sk-old... as the secret on GitHub
  → Investigate why sk-new... doesn't work
  → When you find the problem, repeat from step 1

This is exactly why it's fundamental: never revoke the previous key before verifying the new one.

Edge case 4: Rotation with multiple repos

If the same key is used in several repos:

# List of repos that use the key
REPOS="owner/repo1 owner/repo2 owner/repo3"

# Update in all of them
for REPO in $REPOS; do
  gh secret set OPENAI_API_KEY --repo "$REPO" --body "sk-proj-new..."
  echo "Updated $REPO"
done

# Verify in all of them
for REPO in $REPOS; do
  gh workflow run ci.yml --repo "$REPO"
  echo "Triggered verification in $REPO"
done

Rotation frequency

Regular rotation (preventive)

Emergency rotation (reactive)

Rotate immediately if:

• ✅ A secret appeared in public logs
• ✅ A secret was committed to the repository
• ✅ An employee with access to the secrets leaves the company
• ✅ The provider reports suspicious access
• ✅ You detect unauthorized requests with your key
• ✅ A dependency was compromised and could read environment variables

Signs that you need to rotate

In the OpenAI Dashboard → Usage:
  - Unusual usage spikes (3am on a Sunday)
  - Requests from unknown IPs
  - Models your app doesn't use (GPT-4 when you only use GPT-4o-mini)
  - Costs that don't match your user volume

A post-rotation verification script

#!/usr/bin/env python3
"""
Verify that an API key works correctly after rotation.
It makes a minimal call to confirm authentication and permissions.
"""
import os
import sys
import json
from datetime import datetime, timezone


def verify_openai_key() -> dict:
    try:
        from openai import OpenAI
    except ImportError:
        return {
            "provider": "openai",
            "status": "error",
            "message": "openai package not installed",
        }

    api_key = os.environ.get("OPENAI_API_KEY", "")
    if not api_key:
        return {
            "provider": "openai",
            "status": "error",
            "message": "OPENAI_API_KEY not set",
        }

    try:
        client = OpenAI(api_key=api_key)
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": "Say 'key verified' in 2 words"}],
            max_tokens=10,
            temperature=0.0,
        )
        return {
            "provider": "openai",
            "status": "ok",
            "message": "Key works correctly",
            "model_used": "gpt-4o-mini",
            "response_preview": response.choices[0].message.content[:50],
            "key_prefix": api_key[:8] + "...",
        }
    except Exception as e:
        error_type = type(e).__name__
        return {
            "provider": "openai",
            "status": "error",
            "message": f"{error_type}: {str(e)[:100]}",
            "key_prefix": api_key[:8] + "...",
        }


def main() -> int:
    print("Post-rotation key verification")
    print("=" * 50)

    result = verify_openai_key()

    status_icon = "PASS" if result["status"] == "ok" else "FAIL"
    print(f"\n[{status_icon}] {result['provider']}: {result['message']}")

    if result.get("key_prefix"):
        print(f"  Key prefix: {result['key_prefix']}")
    if result.get("model_used"):
        print(f"  Model: {result['model_used']}")
    if result.get("response_preview"):
        print(f"  Response: {result['response_preview']}")

    report = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "verification": result,
    }

    report_path = "results/key-verification.json"
    os.makedirs("results", exist_ok=True)
    with open(report_path, "w") as f:
        json.dump(report, f, indent=2)
    print(f"\nReport saved to {report_path}")

    return 0 if result["status"] == "ok" else 1


if __name__ == "__main__":
    sys.exit(main())

Expected output (valid key)

Post-rotation key verification
==================================================

[PASS] openai: Key works correctly
  Key prefix: sk-proj-...
  Model: gpt-4o-mini
  Response: key verified

Report saved to results/key-verification.json

Expected output (invalid key)

Post-rotation key verification
==================================================

[FAIL] openai: AuthenticationError: Incorrect API key provided
  Key prefix: sk-proj-...

Report saved to results/key-verification.json

In the workflow

      - name: Verify API key after rotation
        run: python scripts/verify_key_rotation.py
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

      - name: Upload verification report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: key-verification
          path: results/key-verification.json

The rotation checklist

Use this checklist every time you rotate a key:

## Rotation of OPENAI_API_KEY — [date]

### Pre-rotation
- [ ] Verify that no workflows are running
- [ ] Note the previous key (first 8 chars): sk-proj-...
- [ ] Identify every repo/environment that uses this key

### Rotation
- [ ] Generate a new key in the OpenAI Dashboard
- [ ] Note the new key (first 8 chars): sk-proj-...
- [ ] Update the secret on GitHub (repo/staging/production)
- [ ] Trigger the verification workflow
- [ ] Confirm that the workflow passes with the new key

### Post-rotation
- [ ] Revoke the previous key in the OpenAI Dashboard
- [ ] Verify that no workflows are failing
- [ ] Update the internal documentation if applicable
- [ ] Record the rotation in the team's log

Emergency rotation: an exposed key

If you detect that a key was exposed:

Minute 0-5: Containment

# 1. Revoke the compromised key IMMEDIATELY in the provider
# OpenAI Dashboard → API Keys → Revoke

# 2. Generate a new key
# OpenAI Dashboard → API Keys → Create new secret key

# 3. Update it on GitHub
gh secret set OPENAI_API_KEY --body "sk-proj-emergency-new..."

# If you have per-environment secrets
gh secret set OPENAI_API_KEY --env staging --body "sk-proj-emergency-new..."
gh secret set OPENAI_API_KEY --env production --body "sk-proj-emergency-new..."

In an emergency, you invert the steps: first you revoke (to stop the damage), then you generate and update (to restore the service).

Minute 5-15: Verification

# Verify that the pipeline works with the new key
gh workflow run ai-quality-gate.yml
gh run watch

# Review the usage in the OpenAI Dashboard
# Are there suspicious requests after the revocation?

Minute 15-60: Investigation

- How was the key exposed? (commit, log, artifact, chat)
- How long was it exposed?
- Were there unauthorized requests? (review the OpenAI Usage)
- How much did the unauthorized requests cost?
- What measures prevent it from happening again?

Troubleshooting

"After rotating, some workflows fail and others pass"

Cause: You updated the repository secret but not the environment secrets (or vice versa).

Solution: Verify every scope:

# Repository secret
gh secret list
# Environment secrets
gh secret list --env staging
gh secret list --env production

Update every scope that uses the key.

"The new key works locally but not in CI"

Cause: It could be a different rate limit, different project permissions, or the key wasn't updated correctly.

Solution: Verify the secret's value (indirectly):

      - name: Debug key format
        run: |
          echo "Key length: ${#OPENAI_API_KEY}"
          echo "Key prefix: ${OPENAI_API_KEY:0:7}..."
          echo "Key has newlines: $(echo "$OPENAI_API_KEY" | wc -l)"
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

A common mistake: pasting the key with a trailing newline. Verify that wc -l shows 1.

"OpenAI reports requests after revoking the key"

Cause: The requests can take a few minutes to propagate in OpenAI's dashboard. If you see requests up to 5 minutes after the revocation, that's normal.

Solution: Wait 15 minutes and check again. If the requests continue, it could be another compromised key — review every active key in your account.

"I rotate the key every 90 days but I forget to do it"

Cause: Manual rotation with no reminders.

Solution: Create a scheduled workflow that reminds you:

name: Key Rotation Reminder
on:
  schedule:
    - cron: "0 9 1 */3 *"  # First day of each quarter, 9am

jobs:
  reminder:
    runs-on: ubuntu-latest
    steps:
      - name: Create rotation reminder issue
        uses: actions/github-script@v7
        with:
          script: |
            await github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: `🔑 API Key Rotation Due — ${new Date().toISOString().slice(0, 7)}`,
              body: `## Quarterly Key Rotation\n\nIt's time to rotate API keys.\n\n### Checklist\n- [ ] OPENAI_API_KEY (repo)\n- [ ] OPENAI_API_KEY (staging)\n- [ ] OPENAI_API_KEY (production)\n\nSee rotation process in Module 4, Capsule 06.`,
              labels: ['maintenance', 'security'],
            });

Exercises

Exercise 1: Simulate a complete rotation

Create a TEST_ROTATION_KEY secret, update its value, verify that a workflow uses it correctly, and document each step.

# Step 1: Create the initial secret
gh secret set TEST_ROTATION_KEY --body "old-key-value-12345"
echo "Created TEST_ROTATION_KEY with old value"

# Step 2: Create a verification workflow
# (Assume a workflow that uses the secret already exists)

# Step 3: Update the secret (this simulates the rotation)
gh secret set TEST_ROTATION_KEY --body "new-key-value-67890"
echo "Updated TEST_ROTATION_KEY with new value"

# Step 4: Verify
gh workflow run test-secrets.yml
gh run watch

# Step 5: Confirm
gh run list --limit 1
echo "Rotation complete"

Rotation of TEST_ROTATION_KEY — 2026-03-08
  Old key prefix: old-key-v...
  New key prefix: new-key-v...
  Updated at: 14:30 UTC
  Verified at: 14:32 UTC (workflow run #42)
  Status: Success

Exercise 2: An automated rotation script

Write a script that automates steps 2-3 of the rotation (update the secret + verify).

#!/usr/bin/env python3
"""
Semi-automated secret rotation.
Updates the secret and triggers a verification workflow.
"""
import subprocess
import sys
import time
import json


def run_cmd(cmd: str) -> tuple[int, str]:
    result = subprocess.run(
        cmd, shell=True, capture_output=True, text=True
    )
    return result.returncode, result.stdout.strip()


def rotate_secret(
    secret_name: str,
    new_value: str,
    environments: list[str] | None = None,
    verification_workflow: str = "ai-quality-gate.yml",
) -> bool:
    print(f"Rotating {secret_name}...")

    targets = environments or ["repository"]
    for target in targets:
        if target == "repository":
            cmd = f'gh secret set {secret_name} --body "{new_value}"'
        else:
            cmd = f'gh secret set {secret_name} --env {target} --body "{new_value}"'

        code, output = run_cmd(cmd)
        if code != 0:
            print(f"  FAIL: Could not update {target}: {output}")
            return False
        print(f"  OK: Updated {target}")

    print(f"\nTriggering verification workflow: {verification_workflow}")
    code, output = run_cmd(f"gh workflow run {verification_workflow}")
    if code != 0:
        print(f"  FAIL: Could not trigger workflow: {output}")
        return False

    print("Waiting for workflow to complete...")
    time.sleep(5)

    code, output = run_cmd("gh run list --limit 1 --json status,conclusion -q '.[0]'")
    if code == 0:
        run_info = json.loads(output)
        print(f"  Status: {run_info.get('status', 'unknown')}")
        print(f"  Conclusion: {run_info.get('conclusion', 'pending')}")

    print("\nRotation steps completed.")
    print("IMPORTANT: Verify the workflow passes before revoking the old key.")
    return True


if __name__ == "__main__":
    if len(sys.argv) < 3:
        print("Usage: python rotate_secret.py SECRET_NAME NEW_VALUE [env1,env2]")
        sys.exit(1)

    name = sys.argv[1]
    value = sys.argv[2]
    envs = sys.argv[3].split(",") if len(sys.argv) > 3 else None

    success = rotate_secret(name, value, envs)
    sys.exit(0 if success else 1)

# Rotate the repository secret
python scripts/rotate_secret.py OPENAI_API_KEY "sk-new-key..."

# Rotate in specific environments
python scripts/rotate_secret.py OPENAI_API_KEY "sk-new-key..." staging,production

Exercise 3: A rotation reminder workflow

Create a scheduled workflow that creates a GitHub Issue reminding you to rotate the keys every 90 days.

# .github/workflows/rotation-reminder.yml
name: Key Rotation Reminder
on:
  schedule:
    - cron: "0 9 1 */3 *"
  workflow_dispatch:

permissions:
  issues: write

jobs:
  create-reminder:
    runs-on: ubuntu-latest
    steps:
      - name: Create rotation issue
        uses: actions/github-script@v7
        with:
          script: |
            const date = new Date();
            const quarter = `Q${Math.ceil((date.getMonth() + 1) / 3)}`;
            const year = date.getFullYear();

            await github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: `Key Rotation Due — ${year} ${quarter}`,
              body: [
                '## Quarterly API Key Rotation',
                '',
                '### Keys to Rotate',
                '- [ ] `OPENAI_API_KEY` (repository)',
                '- [ ] `OPENAI_API_KEY` (staging environment)',
                '- [ ] `OPENAI_API_KEY` (production environment)',
                '',
                '### Process',
                '1. Generate new key in OpenAI Dashboard',
                '2. Update secret: `gh secret set OPENAI_API_KEY`',
                '3. Trigger verification: `gh workflow run ai-quality-gate.yml`',
                '4. Verify workflow passes',
                '5. Revoke old key in OpenAI Dashboard',
                '',
                '### Documentation',
                'See Module 4, Capsule 06 for the complete rotation process.',
              ].join('\n'),
              labels: ['maintenance', 'security'],
            });

            console.log('Rotation reminder issue created');

Key Rotation Due — 2026 Q2

## Quarterly API Key Rotation

### Keys to Rotate
- [ ] OPENAI_API_KEY (repository)
- [ ] OPENAI_API_KEY (staging environment)
- [ ] OPENAI_API_KEY (production environment)
...

Exercise 4: Document an emergency rotation

Write the runbook (step by step) for an emergency rotation when you detect that your OPENAI_API_KEY was exposed in a public commit.

# Runbook: Emergency Key Rotation — OPENAI_API_KEY Exposed

## Severity: CRITICAL
## Expected time: 15 minutes

### Immediate Actions (0-5 min)

1. **Revoke the exposed key** (do this FIRST, not last)

2. **Generate replacement key**

3. **Update all locations**
```bash
gh secret set OPENAI_API_KEY --body "sk-new-key..."
gh secret set OPENAI_API_KEY --env staging --body "sk-new-key..."
gh secret set OPENAI_API_KEY --env production --body "sk-new-key..."

</details>

---

## Summary

- ✅ **4 steps:** generate a new key → update the secret → verify the pipeline → revoke the previous key
- ✅ **Order matters:** never revoke before verifying — you need an overlap window
- ✅ **In-progress workflows** use the key they had when they started — wait for them to finish
- ✅ **Regular rotation:** every 90 days for LLM providers' API keys
- ✅ **Emergency rotation:** revoke first, generate afterward, verify, document
- ✅ **Post-rotation verification:** a script that makes a minimal call to confirm the key works
- ✅ **Automated reminders:** a scheduled workflow that creates quarterly rotation issues
- ✅ **A checklist:** use a documented checklist for every rotation, don't improvise
- ✅ **If a key gets exposed:** revoke immediately, then generate a new one, then investigate

---

## Additional resources

1. [OpenAI — API Key Management](https://platform.openai.com/api-keys) — Creating, revoking, and managing API keys
2. [GitHub — Secret Scanning](https://docs.github.com/en/code-security/secret-scanning/about-secret-scanning) — Automatic detection of secrets in repos
3. [NIST SP 800-57 — Key Management](https://csrc.nist.gov/publications/detail/sp/800-57-part-1/rev-5/final) — The standard for managing cryptographic keys
4. [BFG Repo-Cleaner](https://rtyley.github.io/bfg-repo-cleaner/) — Cleaning sensitive data from Git's history
5. [GitGuardian](https://www.gitguardian.com/) — Real-time detection of exposed secrets
6. [Anthropic — API Key Security](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching) — Anthropic's best practices for key management