Module 3: Configuration Secrets Health And Autoscaling

4. Hands-on: `ConfigMap` and `Secret` for `andes-cargo-status-api`

Next →

Description

This is the lesson where andes-cargo-status-api receives, for the first time in this guide, the configuration it's been missing since Module 2: a ConfigMap with the address where the LocalStack endpoint would live, the Secret with the dummy credentials, both mounted on the Deployment via envFrom, and verified with no image rebuild, not even once. You're also going to confirm something honest: the /shipments/<id> error doesn't disappear — it changes shape, because no module in this guide installs LocalStack inside the cluster (the data layer stays outside its $0 scope). Everything that follows really ran against andes-cargo-cluster.

Connection to the module

This lesson brings together lessons 2 and 3's two conceptual pieces on top of the real Deployment Module 2 left behind. It's the first time you see, with real evidence, a ConfigMap/Secret's central promise: changing a service's configuration without touching its image.

Before starting: confirm Module 2's state

kubectl get all -n andes-cargo

What to expect (if your lab is still in the same state Module 2 left it in; Pod names, IPs, and AGE are your variable values):

NAME                                          READY   STATUS    RESTARTS   AGE
pod/andes-cargo-status-api-56856576d4-2slnk   1/1     Running   0          19m
pod/andes-cargo-status-api-56856576d4-7ztk9   1/1     Running   0          22m
pod/andes-cargo-status-api-56856576d4-vjc9k   1/1     Running   0          22m

NAME                         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/status-api-service   ClusterIP   10.96.78.1   <none>        80/TCP    21m

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/andes-cargo-status-api   3/3     3            3           22m

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/andes-cargo-status-api-56856576d4   3         3         3       22m

If your cluster doesn't show this, go back to Module 2, lesson 8, before continuing.

Step 1 — The real ConfigMap: andes-cargo-status-api-config

# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: andes-cargo-status-api-config
  namespace: andes-cargo
data:
  DYNAMODB_ENDPOINT_URL: "http://localstack.localstack.svc.cluster.local:4566"
  AWS_REGION: "us-east-1"
  SHIPMENTS_TABLE_NAME: "Shipments"

Three keys, each corresponding to one of the three non-sensitive variables app.py reads (Module 1, lesson 3, already previewed the full contract). Notice DYNAMODB_ENDPOINT_URL's value: it's not just the DNS name (localstack.localstack.svc.cluster.local:4566), it's the full URL with scheme (http://...) — because that's how boto3 expects the endpoint_url parameter in app.py (line 58, Module 6 of aws-serverless-and-containers-guide). The DNS name itself follows the same <service>.<namespace>.svc.cluster.local pattern you already know from Module 2, lesson 6 — pointing to a localstack Service in a localstack namespace that doesn't exist, and isn't going to exist in any module of this guide: installing LocalStack (or wiring up a real AWS account) is outside this Kubernetes lab's $0 scope. This ConfigMap leaves the correct address written for the day someone does wire up that data layer — not a promise this lab is going to install it.

kubectl apply -f configmap.yaml

What to expect:

configmap/andes-cargo-status-api-config created

Step 2 — The real Secret: andes-cargo-status-api-secrets

# secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: andes-cargo-status-api-secrets
  namespace: andes-cargo
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: "test"
  AWS_SECRET_ACCESS_KEY: "test"

Notice stringData, not data: it's a shortcut Kubernetes offers to declare a Secret in YAML with readable plain-text values — more comfortable to write and review in a code review than computing base64 by hand — which kube-apiserver itself automatically encodes when saving the object. The final result, once applied, is identical to if you'd computed the base64 yourself with data.

kubectl apply -f secret.yaml

What to expect:

secret/andes-cargo-status-api-secrets created

Confirm Kubernetes did the automatic conversion:

kubectl get secret andes-cargo-status-api-secrets -n andes-cargo -o yaml

What to expect (literal — creationTimestamp/resourceVersion/uid are your variable values; data is literal, because test in base64 always produces the same result, as you already confirmed in lesson 3):

apiVersion: v1
data:
  AWS_ACCESS_KEY_ID: dGVzdA==
  AWS_SECRET_ACCESS_KEY: dGVzdA==
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{},"name":"andes-cargo-status-api-secrets","namespace":"andes-cargo"},"stringData":{"AWS_ACCESS_KEY_ID":"test","AWS_SECRET_ACCESS_KEY":"test"},"type":"Opaque"}
  creationTimestamp: "2026-08-14T19:55:40Z"
  name: andes-cargo-status-api-secrets
  namespace: andes-cargo
  resourceVersion: "4069"
  uid: d5ca03dc-c99b-44ca-9981-6b6f91a97617
type: Opaque

You wrote stringData: AWS_ACCESS_KEY_ID: "test", and Kubernetes saved data: AWS_ACCESS_KEY_ID: dGVzdA== — the same conversion from lesson 3, done automatically by kube-apiserver the moment the manifest gets applied.

Step 3 — Mount both on the Deployment, via envFrom

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: andes-cargo-status-api
  namespace: andes-cargo
  labels:
    app: andes-cargo-status-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: andes-cargo-status-api
  template:
    metadata:
      labels:
        app: andes-cargo-status-api
    spec:
      containers:
        - name: andes-cargo-status-api
          image: andes-cargo-status-api:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          envFrom:
            - configMapRef:
                name: andes-cargo-status-api-config
            - secretRef:
                name: andes-cargo-status-api-secrets

The only new field versus Module 2's deployment.yaml is envFrom, with two references — one to each object you just created. When kubelet starts a container with this Deployment, it's going to read every key from andes-cargo-status-api-config and andes-cargo-status-api-secrets, and inject them as environment variables, with no need for you to list each one separately with env.

kubectl apply -f deployment.yaml

What to expect:

deployment.apps/andes-cargo-status-api configured

Since template.spec changed, Kubernetes triggers a real RollingUpdate — the same mechanism you're going to study in depth in this guide's Module 5 (deployment strategies) — it creates new Pods with the updated template, and only then removes the old Pods.

kubectl rollout status deployment/andes-cargo-status-api -n andes-cargo --timeout=60s

What to expect (literal, executed):

Waiting for deployment "andes-cargo-status-api" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 1 old replicas are pending termination...
deployment "andes-cargo-status-api" successfully rolled out

Notice something important, answering lesson 2's "Common mistakes": the old Pods (with no envFrom) never updated their environment variables live — Kubernetes replaced them with new Pods, one at a time, each one starting with the correct configuration from the very first second of life. That's the real mechanism behind "a ConfigMap/Secret change requires recreating the Pods."

Step 4 — Verify: the environment variables, with no image rebuild whatsoever

The andes-cargo-status-api:latest image is exactly the same one Module 1 loaded — you didn't rebuild it, didn't run docker build again. Confirm that, even so, the environment variables are there:

kubectl get pods -n andes-cargo -l app=andes-cargo-status-api -o wide

What to expect (Pod names, IPs, and AGE are your variable values — the Deployment's prefix and the two-node pattern are literal):

NAME                                      READY   STATUS    RESTARTS   AGE   IP           NODE                          NOMINATED NODE   READINESS GATES
andes-cargo-status-api-585bd9599d-27btt   1/1     Running   0          21s   10.244.1.5   andes-cargo-cluster-worker2   <none>           <none>
andes-cargo-status-api-585bd9599d-d6bsj   1/1     Running   0          20s   10.244.1.6   andes-cargo-cluster-worker2   <none>           <none>
andes-cargo-status-api-585bd9599d-x4lk9   1/1     Running   0          20s   10.244.2.7   andes-cargo-cluster-worker    <none>           <none>

The ReplicaSet's hash suffix changed (585bd9599d instead of Module 2's 56856576d4) — exactly as expected: by changing template.spec (adding envFrom), Kubernetes computed a new hash, and with it, a new ReplicaSet. Use your own in the following commands.

Use kubectl exec to confirm, from inside a real Pod, that all five variables arrived correctly:

kubectl exec andes-cargo-status-api-585bd9599d-27btt -n andes-cargo -- printenv | grep -E "DYNAMODB|AWS_|SHIPMENTS" | sort

What to expect (literal — substitute your own Pod name):

AWS_ACCESS_KEY_ID=test
AWS_REGION=us-east-1
AWS_SECRET_ACCESS_KEY=test
DYNAMODB_ENDPOINT_URL=http://localstack.localstack.svc.cluster.local:4566
SHIPMENTS_TABLE_NAME=Shipments

Five variables, none declared by hand in the Deployment, none baked into the image — the three from the ConfigMap and the two from the Secret, injected by Kubernetes when the container starts. This is direct proof of lessons 2 and 3's promise: changing configuration without touching the Dockerfile.

Step 5 — This point in the guide's honest limit, again: /shipments/<id>

With the configuration mounted, try again the endpoint Module 2 left failing:

kubectl port-forward svc/status-api-service -n andes-cargo 8080:80

In another terminal:

curl -i -s http://localhost:8080/health

What to expect (no changes from Module 2 — /health never depended on any environment variable):

HTTP/1.1 200 OK
Server: Werkzeug/3.1.8 Python/3.13.15
Date: Fri, 14 Aug 2026 19:56:25 GMT
Content-Type: application/json
Content-Length: 51
Connection: close

{"service":"andes-cargo-status-api","status":"ok"}

Now the endpoint that does depend on the new configuration:

curl -i -s http://localhost:8080/shipments/4471

What to expect (literal, executed — still not 200, and still the correct behavior at this point in the guide):

HTTP/1.1 500 INTERNAL SERVER ERROR
Server: Werkzeug/3.1.8 Python/3.13.15
Date: Fri, 14 Aug 2026 19:56:50 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 265
Connection: close

<!doctype html>
<html lang=en>
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>

Still 500 — but notice something important: this time the request took about 25 seconds to respond (19:56:25 to 19:56:50), it wasn't instant like Module 2's NoCredentialsError. That detail already hints the cause changed. Check the logs to confirm it:

kubectl logs andes-cargo-status-api-585bd9599d-27btt -n andes-cargo --tail=40

What to expect (literal — the exact Pod name is your variable value, the traceback is identical if your Pod handled the request):

urllib3.exceptions.NameResolutionError: AWSHTTPConnection(host='localstack.localstack.svc.cluster.local', port=4566): Failed to resolve 'localstack.localstack.svc.cluster.local' ([Errno -2] Name or service not known)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
  File "/app/app.py", line 27, in get_shipment
    response = dynamodb.get_item(
        TableName=TABLE_NAME,
        Key={"shipmentId": {"S": shipment_id}},
    )
  File "/usr/local/lib/python3.13/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.13/site-packages/botocore/httpsession.py", line 493, in send
    raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://localstack.localstack.svc.cluster.local/"
127.0.0.1 - - [14/Aug/2026 19:56:50] "GET /shipments/4471 HTTP/1.1" 500 -

The cause changed, exactly as this guide predicted in lesson 1: it's no longer NoCredentialsError (Module 2's error, which happened before any network connection attempt, at request-signing time). Now boto3 does have credentials — test/test, mounted by the Secret — and it does try connecting to DYNAMODB_ENDPOINT_URL, but CoreDNS (the cluster's internal name server, Module 1, lesson 5) can't resolve localstack.localstack.svc.cluster.local because no localstack namespace or Service with that name exists, and no module in this guide is going to create one — nothing runs there. NameResolutionError first, wrapped in EndpointConnectionError after botocore exhausts its automatic retries (the reason for the ~25-second wait before the 500).

       WHAT CHANGED BETWEEN MODULE 2 AND THIS LESSON (M3's honesty)

  Module 2, lesson 7         No ConfigMap/Secret     NoCredentialsError
                                                       (fails at SIGNING, instant)

  Module 3, lesson 4          With ConfigMap/Secret    NameResolutionError /
  (this lesson)                                        EndpointConnectionError
                                                        (fails at CONNECTING, ~25s of retries)

  What's still missing, and why it stays missing:
    - LocalStack running inside the cluster itself      ──▶  outside this guide's $0 scope
    - /health (no external dependencies)                 ──▶  already responds 200, and remains
                                                                the signal we use

Stop the port-forward with Ctrl+C before continuing.

Common mistakes

Expecting /shipments/<id> to work now that there's ConfigMap/Secret, and spending time "debugging" a non-problem (expectation, already anticipated in lesson 1). What happens: someone sees this lesson's 500 and starts reviewing configmap.yaml/secret.yaml looking for a typo, not realizing the behavior is exactly what's expected. How to spot it: if you compare DYNAMODB_ENDPOINT_URL's value letter by letter more than once, looking for an error that doesn't exist. How to fix it: this lesson's "This point in the guide's honest limit" section explains the exact cause — there's no Service named localstack in the cluster, and no module in this guide is going to create one — and it's not a mistake on your part: the data layer stays outside this lab's $0 scope by design.

Not noticing the change in response time (~25 seconds) and not investigating the cause (observation). What happens: someone sees the same 500 code as in Module 2 and assumes it's exactly the same error, without noting how long the response took this time. Why it happens: both cases end in 500, and at a glance a curl that takes 25 seconds looks the same as an instant one if you don't time it. How to spot it: compare the request's timestamp (Date in the response header) against when you launched it. How to fix it: an instant 500 almost always indicates a failure happening before any network attempt (like NoCredentialsError); a 500 that takes several seconds almost always indicates network retries running out — the clue this lesson used to diagnose the change in cause with no guessing, just reading kubectl logs.

Forgetting the ReplicaSet changed hash, and looking for old Pods that no longer exist (workflow, the same pattern from Module 2). What happens: someone copies a Pod name from an earlier command in this same lesson (or from a previous session) and kubectl exec/kubectl logs fail with NotFound. How to spot it: the error message explicitly mentions the Pod doesn't exist. How to fix it: every time you change a Deployment's template.spec — as this lesson did by adding envFrom — Kubernetes creates a new ReplicaSet with new-named Pods. Run kubectl get pods -n andes-cargo -l app=andes-cargo-status-api to confirm the current names before any command that depends on a specific name.

Exercises

Exercise 1 — Reconstruct the whole flow from memory. Without going back to the lesson, list the five steps, in order, from creating the ConfigMap to confirming /shipments/<id>'s new error.

See solution
  1. Create configmap.yaml with DYNAMODB_ENDPOINT_URL/AWS_REGION/SHIPMENTS_TABLE_NAME, and apply it.
  2. Create secret.yaml with stringData for AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY, apply it, and confirm the automatic conversion to base64 data.
  3. Add envFrom (with both references) to deployment.yaml, apply it, and wait for the RollingUpdate with kubectl rollout status.
  4. Verify with kubectl exec ... -- printenv that all five variables reached a real Pod, with no image rebuild.
  5. Confirm with curl/kubectl logs that /health stays at 200 and that /shipments/<id> changed from NoCredentialsError to NameResolutionError/EndpointConnectionError.

Exercise 2 — Explain why the ReplicaSet changed hash. A colleague asks why, after only adding envFrom to deployment.yaml, every Pod's name changed completely. Explain it to them in two or three sentences.

See solution

A reasonable answer: "The ReplicaSet's hash is computed from template.spec's complete content — including envFrom, which is part of that template. By changing that field, Kubernetes computed a new hash, created a new ReplicaSet with that hash, and migrated Pods from the old ReplicaSet to the new one with a RollingUpdate — the same mechanism you already saw in Module 2 when you scaled the replica count, except there the hash didn't change because replicas isn't part of template.spec."

Exercise 3 — Predict what would happen if you only added the Secret, without the ConfigMap. If this lesson had mounted only andes-cargo-status-api-secrets (with AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY), but not andes-cargo-status-api-config, what error would you expect to see when requesting /shipments/4471? Justify your answer with what you know about app.py.

See solution

Without DYNAMODB_ENDPOINT_URL in the environment, os.environ.get("DYNAMODB_ENDPOINT_URL") would return None (there's no default value declared for that variable in app.py, unlike SHIPMENTS_TABLE_NAME/AWS_REGION, which do have a default). With endpoint_url=None, boto3 would use its default behavior: try connecting to AWS's real endpoint (dynamodb.us-east-1.amazonaws.com), not LocalStack. With the dummy test/test credentials, that request would fail in a way different from the two seen so far — probably a real AWS authentication error, or a network timeout if your lab doesn't have internet access configured that way — demonstrating that both pieces (ConfigMap and Secret) are needed together, neither substitutes for the other.

Summary and next step

This lesson mounted, for the first time in this guide, real configuration on andes-cargo-status-api: the andes-cargo-status-api-config ConfigMap (endpoint, region, table name) and the andes-cargo-status-api-secrets Secret (dummy credentials), both via envFrom, confirmed with kubectl exec ... -- printenv with no image rebuild whatsoever. /health kept responding 200 with no change. /shipments/4471 still didn't respond 200 — but the error changed shape, from NoCredentialsError (Module 2, instant failure at signing) to NameResolutionError/EndpointConnectionError (this lesson, failure after ~25 seconds of connection retries) — proof the configuration arrived, even though LocalStack never comes to exist inside the cluster: the data layer stays outside this guide's $0 scope, and /health remains the signal that validates the rest of the path.

Before moving on you should be able to: explain the difference between stringData and data in a Secret manifest; verify environment variables inside a Pod with no image rebuild; and diagnose, by reading a traceback, whether a /shipments/<id> failure is from missing credentials or from a missing service to connect to.

Next lesson: liveness, readiness, and startup probes. With configuration resolved, the module moves to lesson 1's second question: how does Kubernetes know whether a running Pod is really ready to handle traffic?

Resources

  1. Kubernetes — Define Environment Variables for a Container — official reference for envFrom, this lesson's central mechanism.
  2. Kubernetes — Secrets: stringData — official documentation for the stringData field used in secret.yaml.
  3. Boto3 — EndpointConnectionError — official reference for botocore's retry behavior, the cause of this lesson's ~25-second wait.
  4. aws-serverless-and-containers-guide (NIEVA), Module 6, lesson 5 — the original app.py, unchanged, whose environment variables this lesson finally completes.
← Previous3. Secrets: why a credential never lives in the imageNext →5. Liveness, readiness, and startup probes