Module 4: Secrets and Environment Management

1. Introduction: Secrets and Environment Management

Overview

Modules 1-3 gave you a complete CI pipeline: workflows with GitHub Actions, automated pytest with matrix testing and caching, and AI-specific checks like prompt regression testing and cost estimation. Your quality gate protects prompt quality and controls costs before every merge. Everything works.

But there's a problem we deliberately dodged in Module 3: the prompt regression testing step has this line:

env:
  OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Where does that secret come from? Who configured it? What happens if someone does echo $OPENAI_API_KEY in a workflow step? What happens if staging and production need different keys? How do you rotate a key without breaking the pipeline?

This module answers all those questions. Secrets management isn't a glamorous topic, but it's one where mistakes cost real money. An OpenAI API key exposed in a public repository can generate $10,000 in charges before you realize it. It's not theoretical — it happens constantly. Automated bots scan GitHub looking for API keys in commits, and they start using them seconds after finding them.

The problem specific to AI systems: Unlike an API key that reads data (a leak is a privacy violation), an LLM provider's API key has direct billing. Every call generates a cost. An attacker with your OpenAI key doesn't "read your data" — they generate thousands of requests that you pay for. The impact is financial and immediate.

Where Are We in the Guide?

Context

This guide has 8 modules organized into 3 phases:

Phase 1: CI Fundamentals (Modules 1-3)
├── Module 1: Introduction to CI/CD and GitHub Actions    ✅ Completed
├── Module 2: Automated Testing in CI                     ✅ Completed
└── Module 3: AI-Specific CI Checks                       ✅ Completed

Phase 2: CD & Deployment Pipelines (Modules 4-6)
├── Module 4: Secrets and Environment Management          ← YOU ARE HERE
├── Module 5: Docker in CI/CD
└── Module 6: Deployment Pipelines

Phase 3: Production Pipelines (Modules 7-8)
├── Module 7: Monitoring, Notifications and Advanced Patterns
└── Module 8: Capstone Project — Production AI Pipeline

Estimated duration of the module: 60-90 minutes.

Where are we headed?

This module opens Phase 2 (CD & Deployment Pipelines). It's the bridge between "I have checks that run in CI" and "I have a pipeline I can deploy safely to different environments." The progression is deliberate:

First you understood CI (module 1) — workflows, jobs, steps, triggers
Then you automated testing (module 2) — pytest, matrix, caching, reports
You added AI-specific checks (module 3) — prompt regression, cost estimation, linting
Now you handle secrets (this module) — secure API keys in CI for the AI checks
Then you build Docker images (module 5) — automated containerization with registry auth
You deploy to staging and prod (module 6) — deployment pipelines with approval gates
You add monitoring and patterns (module 7) — notifications, scheduled workflows
You integrate everything (module 8) — a complete commit-to-production pipeline

Context: What connects this module with the previous one?

Module	What you learned/will learn
Module 1	CI/CD concepts, GitHub Actions basics, first workflow
Module 2	pytest in CI, matrix testing, caching, reports, timeouts
Module 3	AI-specific checks: prompt regression, cost estimation, linting
Module 4	Secrets management, environment variables, per-environment configs
Module 5	Docker in CI/CD
Module 6	Deployment pipelines
Module 7	Advanced patterns
Module 8	Capstone project

The progression

Module 3:  AI checks that need API keys
    ↓
Module 4:  How to handle those API keys safely (this module)
    ↓
Module 5:  Docker with credentials for registries
    ↓
Module 6:  Deploy with per-environment secrets

Module 3 left a loose end: ${{ secrets.OPENAI_API_KEY }} appeared in the workflows but we never explained how to configure it or how to protect it. This module ties up that loose end and prepares you for modules 5-6, where you'll need credentials for Docker registries and deployment targets.

The real problem: API keys in CI

Scenario 1: The naive hardcoding

# ❌ NEVER do this
- name: Run prompt regression
  run: python scripts/evaluate_prompts.py
  env:
    OPENAI_API_KEY: "sk-proj-abc123..."

The key ends up in the code. Anyone with access to the repo sees it. If the repo is public, bots find it in seconds. If the repo is private, any collaborator can copy it.

Scenario 2: The accidental echo

# ❌ This is dangerous too
- name: Debug environment
  run: |
    echo "API key: $OPENAI_API_KEY"
    python scripts/evaluate_prompts.py
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

GitHub Actions masks secrets in the logs (it replaces them with ***), but that only works if the secret appears exactly as it was defined. If you pass it through a base64 encode, a partial echo, or a script that logs it in a transformed way, the masking may not work.

Scenario 3: The unexpected invoice

Monday: A developer pushes code with a hardcoded OPENAI_API_KEY
Tuesday: A bot detects the key on GitHub
Wednesday: 2.3 million requests generated with your key
Thursday: You get an email from OpenAI: "Your $8,742.00 invoice is ready"
Friday: You revoke the key, but the charges have already been generated

This is not hypothetical. It's one of the most common incidents in the AI engineering community.

What this module covers

Capsule 02: GitHub Secrets

The fundamentals. How to create repository secrets, understand log masking, and use ${{ secrets.X }} correctly in workflows. The difference between repository secrets, environment secrets, and organization secrets. Why you must never echo a secret.

Capsule 03: Environment Variables in Workflows

The distinction between env and secrets. Environment variables at the workflow, job, and step level. Precedence when there are conflicts. When to use env (non-sensitive configuration) vs secrets (credentials).

Capsule 04: Per-Environment Secrets

GitHub Environments: staging and production with different API keys. Protection rules on environments. Required reviewers for production deployments. How the workflow selects the right environment.

Capsule 05: Env Files and .env in CI

How to generate .env files dynamically from secrets in CI. The pattern of creating env files in workflow steps without committing the file. .env in .gitignore — always.

Capsule 06: Secret Rotation

The complete process: generate a new key, update the secret on GitHub, verify that the pipeline works, revoke the previous key. Edge cases: what happens if a workflow is running with the previous key while you update it. The recommended rotation frequency.

Capsule 07: OIDC — Authentication Without Static Secrets

A conceptual explanation of OpenID Connect. Why identity-based authentication is the future. How it would work with AWS/GCP. It's not a complete implementation — it's context for guide #17 (Cloud Infrastructure).

Goal of the module

By completing this module you will be able to:

✅ Configure GitHub Secrets (repository, environment, organization) and use them in workflows
✅ Distinguish between env and secrets in GitHub Actions and choose correctly
✅ Configure GitHub Environments with different secrets for staging and production
✅ Generate .env files dynamically in CI from secrets
✅ Implement a secret rotation strategy without downtime
✅ Understand OIDC conceptually and know when to apply it
✅ Prevent secret leaks in logs, artifacts, and outputs

The professional goal

When your team has a CI/CD pipeline that needs API keys from OpenAI, Anthropic, or any other provider, you'll know: (1) how to store them safely on GitHub, (2) how to use them in workflows without exposing them, (3) how to handle different keys for staging and production, and (4) how to rotate them without breaking anything. That's professional secrets management.

Prerequisites

✅ Module 3 completed: You built the AI Quality Gate with prompt regression and cost estimation
✅ Working GitHub Actions: You know how to create workflows, you understand jobs, steps, secrets syntax
✅ An OpenAI API key: You need a real key to practice (no mocks)
✅ Access to the repository's Settings: To configure secrets and environments

The pipeline you're going to secure

Your current pipeline (after Module 3) has this structure:

name: AI Quality Gate
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ruff check src/ tests/ scripts/
      - run: mypy src/ --ignore-missing-imports

  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pytest tests/ -v --tb=short

  prompt-regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python scripts/evaluate_prompts.py
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}  # ← Where does this come from?

  cost-estimation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python scripts/estimate_cost.py

The prompt-regression job needs OPENAI_API_KEY. The cost-estimation job runs locally with tiktoken (no API). The lint and test jobs don't need secrets.

What this module adds

name: Secure AI Quality Gate
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  lint:
    # No changes — it doesn't need secrets

  test:
    # No changes — it doesn't need secrets

  ai-checks-staging:
    runs-on: ubuntu-latest
    environment: staging           # ← GitHub Environment
    steps:
      - run: python scripts/evaluate_prompts.py
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}  # ← Staging's key
          APP_ENV: staging

  ai-checks-production:
    runs-on: ubuntu-latest
    environment: production        # ← A different environment
    needs: [ai-checks-staging]    # ← Only after staging
    steps:
      - run: python scripts/evaluate_prompts.py
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}  # ← Production's key
          APP_ENV: production

The difference: secrets configured securely, separate environments, different keys for staging and production, and protection against leaks.

Technical setup

What you need

# An OpenAI API key (real, not a mock)
# You get it at: https://platform.openai.com/api-keys

# Access to your repository on GitHub
# Settings → Secrets and variables → Actions
# Settings → Environments

# GitHub CLI (optional but recommended)
gh --version
# Expected output: gh version 2.x.x

The module's file structure

This module doesn't add new files to the project — it modifies the repository's configuration on GitHub and adjusts the existing workflows. The main changes are:

your-ai-project/
├── .github/
│   └── workflows/
│       └── ai-quality-gate.yml    # Modify it to use secrets correctly
├── .gitignore                     # Add .env if it isn't there
├── .env.example                   # A template of the variables (WITHOUT real values)
└── scripts/
    └── verify_secrets.py          # Verification script (new)

Quick verification

Before starting, verify that you can access the secrets configuration:

# Verify access to the repo with the gh CLI
gh repo view --json name,owner
# Expected output: {"name":"your-repo","owner":{"login":"your-username"}}

# List the existing secrets (if any)
gh secret list
# Expected output: (an empty list, or the secrets you already have)

If gh secret list works without errors, you have the necessary permissions.

The levels of secrets management

Level 1: Repository Secrets (the basics)

GitHub → Settings → Secrets → Actions → New repository secret
  Name: OPENAI_API_KEY
  Value: sk-proj-abc123...

Workflow:
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Enough for a personal project or a small team. One secret, one value, the whole repo uses it.

Level 2: Per-Environment Secrets (professional)

GitHub → Settings → Environments → "staging"
  Secret: OPENAI_API_KEY = sk-staging-xxx...

GitHub → Settings → Environments → "production"
  Secret: OPENAI_API_KEY = sk-prod-yyy...

Workflow:
  job-staging:
    environment: staging
    # Uses sk-staging-xxx...

  job-production:
    environment: production
    # Uses sk-prod-yyy...

Staging uses a key with a limited budget. Production uses the real key. If staging generates excessive costs, production isn't affected.

Level 3: OIDC (enterprise)

There are no static secrets.
The workflow authenticates with an identity token.
The cloud provider verifies the identity and grants temporary access.

No keys to rotate, no secrets that can leak. It's the future, but it requires configuration on the cloud provider. We cover it conceptually in capsule 07.

What this module does NOT cover

❌ Vault solutions (HashiCorp Vault, AWS Secrets Manager): They're enterprise tools that are out of scope for this guide. GitHub Secrets is enough for 90% of AI projects.
❌ A complete OIDC implementation: Capsule 07 is conceptual. The real implementation with AWS/GCP is part of guide #17 (Cloud Infrastructure).
❌ Secrets in Kubernetes: If you deploy to K8s, K8s secrets are a separate topic — guide #17 covers it.
❌ Compliance frameworks (SOC2, HIPAA): The patterns we teach are good practices, but we don't cover specific compliance.
❌ Multi-provider key management: We focus on OpenAI as the example. The same patterns apply to Anthropic, Google AI, etc.

Analogy: The key to your house

Imagine you need to give a cleaning service (the CI workflow) access to your house. There are three ways:

Level 1: Give them a copy of the key (Repository Secret). It works, but if they lose it, anyone can get in. And you only have one key for everything — the same key opens the front door, the garage, and the safe.

Level 2: Give them different keys for different areas (Per-Environment Secrets). One key for the kitchen (staging), another for the rest of the house (production). If they lose the staging key, you only compromise the kitchen.

Level 3: A facial recognition system (OIDC). There's no physical key to lose. The service identifies itself, the system verifies who it is, and grants it temporary access to the areas it needs. When it's done, the access is revoked automatically.

All three levels are valid — the one you choose depends on the sensitivity of what you're protecting and the maturity of your operation. This module teaches you all three.

Quick self-assessment

Before starting, verify that you have the necessary context:

How do you reference a secret in a GitHub Actions workflow? (Module 3)
What does continue-on-error: true do on a step? (Module 2)
What are artifacts in GitHub Actions and what are they used for? (Module 2)
What is prompt regression testing and why does it need an API key? (Module 3)
What's the difference between a job and a step? (Module 1)

If any question sounds completely new, review the previous modules before continuing.

Evidence of success

By the end of this module, you should be able to:

Create a repository secret on GitHub and use it in a workflow
Explain why echo ${{ secrets.X }} shows *** in the logs
Configure two GitHub Environments (staging and production) with different secrets
Generate a .env file dynamically in a workflow step
Describe the process of rotating an API key without downtime
Explain conceptually what OIDC is and why it eliminates the need for static secrets
Verify that no secret gets exposed in your pipeline's logs

If you tick every check → you're ready for Module 5.

Additional resources

GitHub Actions — Encrypted Secrets — Official documentation of secrets in GitHub Actions
GitHub Environments — Configuring environments for deployment
OpenAI API Key Best Practices — OpenAI's security recommendations
GitHub Actions OIDC — OIDC authentication in GitHub Actions
GitGuardian — State of Secrets Sprawl — The annual report on secrets exposed on GitHub
OWASP Secrets Management Cheat Sheet — OWASP's best practices for secrets

← Previous8. Project — AI Quality Gate Next →2. GitHub Secrets