Module 3: Configuration Secrets Health And Autoscaling

3. Secrets: why a credential never lives in the image

Next →

Description

cloud-security-and-guardrails-guide (Module 3, lesson 2) already built, with real evidence, the complete case for why a .secrets file on disk — even perfectly gitignored since the first commit — still isn't acceptable the day credentials stop being dummies: no access control of its own, no record of who read it, no assisted rotation mechanism whatsoever. This lesson picks up exactly that same argument, from Kubernetes' angle: a ConfigMap — the previous lesson's object — has the same underlying problem as that .secrets file, and Kubernetes offers a different object for the sensitive case. But this lesson also includes the honesty this guide's design promises: a Kubernetes Secret is not encrypted by default, and anyone who treats it as if it were is making a mistake that can cost them dearly.

Connection to the module

This lesson completes the second of this module's three pieces. Lesson 4 really mounts both lesson 2's ConfigMap and this lesson's Secret on andes-cargo-status-api, closing the externalized-configuration loop lesson 1 promised.

The same antipattern, seen from Kubernetes

Pick back up the .secrets file cicd-and-gitops-on-aws-guide built and cloud-security-and-guardrails-guide analyzed in depth:

AWS_ACCESS_KEY_ID=test
AWS_SECRET_ACCESS_KEY=test

Two lines of plain text, on a physical disk. That guide identified three exact reasons why that design — even perfectly gitignored — would stop being acceptable the day those credentials became real:

  1. No access control of its own. Any process with read access to the filesystem can open the file with cat, with no additional condition.
  2. No record of who read it, or when. Opening a text file generates no observable event.
  3. No assisted rotation mechanism. Changing the value means editing the file by hand and trusting that everything using it finds out about the change in time.

A Kubernetes ConfigMap — lesson 2's object — inherits exactly the same problem: its values live in plain text, visible with a simple kubectl get configmap -o yaml, with no marker distinguishing "this is an endpoint" from "this is a credential." If you put AWS_ACCESS_KEY_ID into the previous lesson's ConfigMap, you'd have, inside etcd (the cluster's database, Module 1, lesson 6), exactly the same antipattern as that .secrets file — just now living inside Kubernetes instead of on a local disk.

What DOES change with a Secret, and what DOESN'T

A Kubernetes Secret object has the same shape as a ConfigMap — key-value pairs, namespace scope, mountable via envFrom or volumeMounts — but with two real differences:

  • base64 encoding, not direct plain text (see the demonstration below — and the important warning that follows).
  • RBAC treats it as a distinct resource. A Kubernetes role can grant read permission over configmaps without granting it over secrets, and vice versa — the first time, in this guide, that the word "secret" has a real, separate access-control consequence. (Module 4's NetworkPolicy and Module 6's Gatekeeper ConstraintTemplate are going to be able to apply specific rules to Secret objects they'd never apply to a ConfigMap.)

What does NOT change, and it needs saying plainly: a Kubernetes Secret is not encrypted by default. base64 is an encoding, not encryption — it's reversible with no key whatsoever, with a single command line. Anyone with read permission on the object (kubectl get secret -o yaml, or direct etcd access) can recover the original value in a second. The only way a Secret is really encrypted at rest is enabling etcd encryption at the cluster level (EncryptionConfiguration, a kube-apiserver setting this guide doesn't enable on kind for lab simplicity, and which on real EKS is resolved differently, with AWS KMS — a topic Module 7 picks up). Without that additional layer, a Secret is, underneath, a ConfigMap with a reversible encoding and stricter access control — not a safe.

Demo: base64 is not encryption, with real evidence

Before creating andes-cargo-status-api's real Secret (that's lesson 4), confirm with your own hands that base64 reverses with no key at all:

echo -n "test" | base64

What to expect (literal, deterministic — base64 is a pure function, the same text always produces the same output):

dGVzdA==

Now reverse that same encoding, with no key or password whatsoever:

echo -n "dGVzdA==" | base64 -d

What to expect:

test

Confirm the same mechanism with a real Kubernetes Secret, created imperatively as a disposable demonstration:

kubectl create secret generic demo-secret -n andes-cargo \
  --from-literal=AWS_ACCESS_KEY_ID=test \
  --from-literal=AWS_SECRET_ACCESS_KEY=test

What to expect:

secret/demo-secret created
kubectl get secret demo-secret -n andes-cargo -o yaml

What to expect (literal — creationTimestamp/resourceVersion/uid are your variable values; the data values are literal, because test in base64 always produces the same result):

apiVersion: v1
data:
  AWS_ACCESS_KEY_ID: dGVzdA==
  AWS_SECRET_ACCESS_KEY: dGVzdA==
kind: Secret
metadata:
  creationTimestamp: "2026-08-14T19:55:26Z"
  name: demo-secret
  namespace: andes-cargo
  resourceVersion: "4043"
  uid: d4fa33b2-98f2-4a34-89e3-68e1aa2b64db
type: Opaque

There it is: kubectl create secret encoded test as dGVzdA== automatically — you didn't have to do it yourself — but anyone with permission to read this object can reverse it effortlessly:

kubectl get secret demo-secret -n andes-cargo -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 -d

What to expect:

test

No key, no password, no extra step — the exact same command pipeline that decoded dGVzdA== manually a few paragraphs back. This isn't a Kubernetes flaw: it's a documented design decision, and the reason RBAC (who can run kubectl get secret) is the real protection, not the encoding itself.

Clean up the demo object:

kubectl delete secret demo-secret -n andes-cargo

What to expect:

secret "demo-secret" deleted from andes-cargo namespace

Why this guide keeps using test/test, with eyes open

This guide is going to create, in lesson 4, a real Secret with exactly the same dummy values aws-serverless-and-containers-guide already used: AWS_ACCESS_KEY_ID=test, AWS_SECRET_ACCESS_KEY=test. It's the same decision, documented the same way that guide documented it: LocalStack doesn't validate that these credentials are real, only that they're present — they're not a secret to protect, they're a placeholder. Using Kubernetes' Secret object to hold them, even so, isn't theater: it's the correct practice, exercised with the correct object, over data whose leaking doesn't matter. The day this same pattern were applied against a real AWS account, the object would be the same Secret — what would change is how serious an RBAC mistake would be, not the mechanics.

Analogy: the recipe on the wall, the safe's combination

A hotel's room-service menu can be posted on the front-desk wall, visible to anyone — it's useful information, with no risk if someone reads it without permission. The hotel safe's combination is a different matter entirely: it doesn't get posted on any wall, it lives somewhere else, with access restricted to whoever really needs it. A ConfigMap is the recipe on the wall — the LocalStack endpoint, the table name, any data whose leaking doesn't matter. A Secret is the safe's combination — but, and this is this lesson's honesty, a safe with the combination written on a paper inside a transparent envelope: anyone with access to the envelope (kubectl get secret permission) can read the combination with no effort whatsoever. The transparent envelope is still better than leaving the combination posted on the same wall as the menu — but don't confuse "separate" with "encrypted."

Common mistakes

Believing a Kubernetes Secret is encrypted, and letting your guard down about who has read access (conceptual, this lesson's most important one). What happens: someone sees the word "Secret" and the fact that values show up encoded in kubectl get -o yaml, and assumes they're protected the same way a hashed password is. Why it happens: the object's name and the fact that you don't see the plain text at a glance invite that conclusion — but this lesson's demonstration proves the opposite with a single command. How to spot it: if you never checked which RBAC has get/list permission over secrets in your cluster, assuming "they're already protected." How to fix it: treat any read permission over secrets with the same seriousness you'd give a read permission over a credentials file on disk — because, in practice, with base64 in the mix, that's exactly what it is.

Confusing "base64" with "obfuscated enough" to paste into a chat or a ticket (discipline). What happens: someone copies data.AWS_SECRET_ACCESS_KEY's value from a kubectl get secret -o yaml and pastes it into a Slack channel or a ticket, reasoning "it's encoded, it can't be read directly." How to spot it: if you search, in your team's message history, for a string ending in == or = (base64's characteristic padding) near the word "secret." How to fix it: any Secret's data value must be treated as plain text for all practical handling purposes — this lesson's demonstration took under a second to reverse it.

Using kubectl create secret without -n andes-cargo and breaking the Deployment's reference (configuration, the same pattern from lesson 2). What happens: the Secret gets created in default instead of andes-cargo, and the Deployment referencing it — in the correct namespace — never finds it, producing an error when trying to mount it. How to spot it: kubectl describe pod shows an event like Error: secret "andes-cargo-status-api-secrets" not found if the Pod tries to start without finding the Secret in its own namespace. How to fix it: lesson 4 uses -n andes-cargo in every command, exactly for this reason — a Secret, same as a ConfigMap, is only visible to Pods in its own namespace.

Exercises

Exercise 1 — Explain the difference between ConfigMap and Secret without saying "one is more secure." In two or three sentences, without using the phrase "more secure" or "encrypted," explain to a colleague the real difference between a Kubernetes ConfigMap and a Secret.

See solution

A reasonable answer: "Both store key-value pairs the same way, with the same ease of reading if you have the right permission — the real difference is that RBAC can grant access to one without granting it to the other, so a Secret lets you separate who can read general configuration from who can read credentials. The base64 encoding doesn't add protection by itself, it's reversible with one command."

Exercise 2 — Reconstruct cloud-security-and-guardrails-guide's three reasons applied to a ConfigMap. Without going back to the corresponding section, explain why putting a credential in a Kubernetes ConfigMap has the same three problems as that sibling guide's .secrets file.

See solution
  1. No additional access control of its own — a ConfigMap has no RBAC separation from other resources of the same type; anyone with general configmaps read permission would see the credential.
  2. No record of who read it — reading a ConfigMap via kubectl get -o yaml doesn't, by itself, generate any audit event different from reading any other ConfigMap.
  3. No assisted rotation mechanism — changing a ConfigMap's value means hand-editing plain text, with no automatic coordination with whoever consumes it.

Exercise 3 — Decode a Secret without using kubectl -o jsonpath. If you had the complete output of kubectl get secret <name> -o yaml copied into a text file, which two Unix commands would you chain to get a specific data field's real value, without using any special kubectl flag?

See solution

grep <key>: file.yaml | awk '{print $2}' | base64 -d — or any equivalent combination that extracts the corresponding line's value and pipes it to base64 -d. The exercise's point is confirming that decoding a Secret doesn't depend on any special Kubernetes tool: any text ending with base64's padding reverses with the same standard Unix command used in this lesson.

Summary and next step

This lesson picked back up, from Kubernetes, the same argument cloud-security-and-guardrails-guide already built against a .secrets file on disk: a plain-text credential, with no access control of its own and no read record, is an antipattern no matter where it lives. A Kubernetes Secret object adds a real RBAC separation versus a ConfigMap — but, with the full honesty this guide promises, it does not add encryption by default: base64 is an encoding reversible in a second, confirmed with real evidence in this lesson. A Secret's real protection is who has permission to read it, not how it's encoded.

Before moving on you should be able to: explain why base64 isn't encryption, with a single-command demonstration; name the real RBAC difference between ConfigMap and Secret; and justify why this guide keeps using dummy credentials (test/test) even inside an object meant for sensitive data.

Next lesson: hands-on, ConfigMap and Secret for andes-cargo-status-api. There you build the real objects — not demonstration ones — and mount them on the Deployment, with no image rebuild, not even once.

Resources

  1. Kubernetes — Secrets — complete official reference; the "Uses for Secrets" section and the explicit note that a Secret isn't encrypted by default are this lesson's technical foundation.
  2. Kubernetes — Encrypting Confidential Data at Rest — official documentation for the real encryption mechanism (etcd's EncryptionConfiguration), named but not enabled in this guide.
  3. cloud-security-and-guardrails-guide (NIEVA), Module 3, lesson 2 — the original .secrets file antipattern, which this lesson picks back up from Kubernetes.
  4. aws-serverless-and-containers-guide (NIEVA), Module 6, lesson 5 — the origin of the dummy test/test credentials, reused with no changes in lesson 4's Secret.
← Previous2. ConfigMaps: separating configuration from the imageNext →4. Hands-on: `ConfigMap` and `Secret` for `andes-cargo-status-api`