Module 3: Configuration Secrets Health And Autoscaling

1. Module introduction: what a Deployment alone doesn't give you

Description

Module 2 ended with a checklist that looked complete: a Deployment with three replicas, a ReplicaSet replacing Pods deleted on purpose, a Service balancing real traffic across all three — and an honest verification that /shipments/4471 still responded 500, because andes-cargo-status-api still had no way to talk to a database. This module starts exactly there, with a question that checklist leaves open on purpose: is three running replicas the same as "production ready"? The short answer, which the rest of this module demonstrates with real evidence, is no — and the three exact reasons why are the three pieces you're going to build here.

Connection to the module

This module takes the same andes-cargo-status-api Deployment Module 2 left behind — healthy, balanced, with three replicas — and adds, in order, the three things it's missing to stop being a lab experiment and start looking like a real production service: externalized configuration (ConfigMap/Secret, lessons 2-4), health verification (liveness/readiness/startup probes, lessons 5-6), and elastic capacity (HorizontalPodAutoscaler, lessons 7-8). None of the three is optional in a real cluster — together, all three are the difference between "it runs" and "it's ready for someone else to depend on it."

The three questions "3 replicas running" doesn't answer

Go back to Module 2's final checklist (kubectl get all -n andes-cargo): a Deployment at 3/3, a Service with three Endpoints, everything Running. That snapshot answers one question — "does the number of Pods I declared match the number of Pods that exist?" — but leaves three others unanswered, each with real consequences the day this system had to carry real traffic:

Question 1 — Where does this service get its configuration from, and what happens when that configuration changes? Today, andes-cargo-status-api has no environment variable configured at all — that's exactly why /shipments/<id> fails with NoCredentialsError (Module 2, lesson 7). But even if it had those variables, where would they live? The naive answer — "inside the Dockerfile, as a fixed ENV" — creates a real problem: every time the LocalStack endpoint changed (from one test environment to another, from kind to a real cluster), the entire image would need rebuilding. Lessons 2-4 of this module resolve this with ConfigMap and Secret: configuration lives outside the image, mounts when the Pod starts, and changes with no docker build at all.

Question 2 — How does Kubernetes know whether a Pod that "is running" is the same as being "ready to handle traffic"? Today, the answer is: it doesn't know, because you never asked it. kubectl get pods tells you STATUS: Running the moment the process inside the container starts — but "the process started" and "the process can meaningfully handle an HTTP request" aren't the same thing, and this guide hasn't yet given Kubernetes any way to tell them apart. If andes-cargo-status-api hung — without crashing, just no longer responding — the Service would keep sending it traffic indefinitely, because nothing tells it the Pod stopped being healthy. Lessons 5-6 resolve this with probes.

Question 3 — What happens if traffic multiplies by ten tomorrow? Today, the answer is: nothing, until someone notices the problem and edits deployment.yaml by hand to raise replicas. Three replicas is a number you decided, once, with no relationship whatsoever to the real load the system receives at any given moment — it can be excessive all night and insufficient during peak hour. Lessons 7-8 resolve this with the HorizontalPodAutoscaler, which adjusts the replica count on its own, based on a real metric.

                  WHAT MODULE 2 LEFT, AND WHAT THIS MODULE ADDS

  ┌─────────────────────────────────────────────────────────────────┐
  │  andes-cargo-status-api          3/3 replicas, balanced             │
  │  status-api-service (ClusterIP)  correct Endpoints                  │
  │  /health                         200 OK                             │
  │  /shipments/<id>                 500 (NoCredentialsError, honest)   │
  └──────────────────────────────┬──────────────────────────────────┘
                                    │
        ┌───────────────────────────┼───────────────────────────┐
        ▼                           ▼                           ▼
  ConfigMap/Secret            probes                      HorizontalPodAutoscaler
  (lessons 2-4)                (lessons 5-6)                (lessons 7-8)
  configuration outside       Kubernetes tells              replicas adjusted
  the image                   "running" from "ready"         by real metric, not
                                                               a fixed number

Analogy: the hotel that already takes guests, but hasn't passed inspection

Module 2 left Andes Cargo's hotel with three occupied rooms and a front desk (Deployment) that guarantees that number never drops. That's real, and it's an achievement — but any hotel knows "having guests" isn't the same as "being certified to operate." Three things are missing, exactly this module's three: a operations manual separate from the building's structure — the room-service menu isn't painted on the wall, it lives in a folder that gets updated without breaking a single brick (ConfigMap/Secret); a health-check routine, not just "someone is registered in the room," but "that person answers if you knock, and if they don't answer after several tries, they need to be replaced" (probes); and a staffing policy that adjusts to real occupancy, not a fixed number of front-desk staff decided once and never revisited, but one that goes up on a long weekend and down midweek, according to real demand (HorizontalPodAutoscaler). No real hotel operates without all three — and neither does any Kubernetes cluster in production.

This module's map

#	Lesson	What it resolves
2	`ConfigMap`: separating configuration from the image	Why the LocalStack endpoint doesn't live in the `Dockerfile`
3	`Secret`: why a credential never lives in the image	The same problem as `ConfigMap`, with the added layer of it being sensitive
4	Hands-on: real `ConfigMap` + `Secret`	Mounted on `andes-cargo-status-api`, with no image rebuild
5	Liveness, readiness, and startup probes	What each probe asks, and what Kubernetes does with the answer
6	Hands-on: real probes, with induced failures	A readiness failure (leaves the Service) and a liveness failure (restarts), really observed
7	`HorizontalPodAutoscaler`: scaling by metrics	`metrics-server` on `kind`, and the HPA over CPU
8	Project: `andes-cargo-status-api` under load	Real load that triggers the HPA, replicas going up and down on their own

By the end of this module, andes-cargo-status-api is going to have the three pieces it was missing — and Module 4 is going to expose that same, now more mature, service to the cluster's external traffic with Ingress.

Common mistakes

Thinking this module "fixes" /shipments/<id> completely (expectation, the most important one to prevent before starting). What happens: someone arrives at this module expecting that, once lesson 4 (ConfigMap/Secret) is done, the /shipments/<id> endpoint finally responds 200 with real data. Why it happens: it's easy to assume "adding the missing configuration" resolves the whole problem in one shot. How to spot it: if you're surprised lesson 4 still shows an error after mounting ConfigMap/Secret. How to fix it: remember this guide's design's explicit honesty — this module's ConfigMap/Secret give andes-cargo-status-api the credentials and the address to look for, but no module in this guide installs LocalStack inside the cluster (the data layer stays outside its $0 scope). The error is going to change shape — from NoCredentialsError to a connection error — but continuing to exist is the correct behavior, and it stays that way through the capstone.

Assuming this module's three pieces are independent of each other (conceptual). What happens: someone treats ConfigMap, probes, and HPA as three unrelated topics, instead of seeing that all three together build the same idea: a service Kubernetes can manage with no constant human intervention. Why it happens: each lesson is taught separately, with its own example. How to spot it: if you can't explain, in one sentence, why all three pieces belong to the same module. How to fix it: think about this lesson's diagram — externalized configuration, health verification, and elastic capacity are the three conditions that, together, make it possible for Kubernetes to reconcile the system's state with no one editing a YAML by hand every time something changes. It's the same idea from Module 2's control loop (lesson 3), applied to three new dimensions.

Skipping Module 2's final checklist before starting here (workflow). What happens: someone starts this module without confirming their cluster is still in the exact state Module 2 left it in — three healthy replicas, Service balancing — and runs into errors later that actually come from a different starting state. How to spot it: if kubectl get all -n andes-cargo doesn't show exactly one andes-cargo-status-api Deployment at 3/3 and one status-api-service Service. How to fix it: before lesson 2, run kubectl get all -n andes-cargo and compare it against Module 2's final checklist, lesson 8. If something doesn't match, go back there before continuing.

Exercises

Exercise 1 — Name the three questions without rereading the lesson. Without going back to the corresponding section, write the three questions "3 replicas running" doesn't answer, and which piece of this module resolves each one.

See solution

Where does this service get its configuration from, and what happens when it changes? — resolved by ConfigMap/Secret (lessons 2-4).
How does Kubernetes know whether a running Pod is ready to handle traffic? — resolved by liveness/readiness/startup probes (lessons 5-6).
What happens if traffic suddenly multiplies? — resolved by the HorizontalPodAutoscaler (lessons 7-8).

Exercise 2 — Explain the hotel analogy to a colleague who didn't read it. In two or three sentences, without using the word "Kubernetes," explain why a hotel with occupied rooms might still not be ready to operate, using this lesson's same three gaps.

See solution

A reasonable explanation: "Having guests isn't the same as being certified to operate. It's missing a service manual that can be updated without remodeling the building, a routine that confirms each guest is still really there (not just that they checked in at some point), and a staffing policy that goes up and down based on how many people are there today, not a fixed number decided once and never revisited."

Exercise 3 — Predict which error you're still going to see in lesson 4. With this lesson's honesty in mind, predict: after lesson 4 mounts ConfigMap and Secret on andes-cargo-status-api, are you going to be able to confirm /shipments/4471 with a real 200? Justify your answer.

See solution

No — you're still going to see an error, though a different type from Module 2's NoCredentialsError. With credentials and the endpoint configured, boto3 is no longer going to fail from missing credentials, but it's going to fail trying to connect: DYNAMODB_ENDPOINT_URL is going to point to a DNS name (localstack.localstack.svc.cluster.local) that doesn't exist inside the cluster, because no module in this guide installs LocalStack there. The error changes shape, but the result — a response that isn't 200 — remains the correct behavior throughout the guide: /health is the real signal, /shipments/<id> stays representative.

Summary and next step

This lesson opened Module 3 with an honest question: the "green" checklist Module 2 left behind — three replicas, balanced, stable Service — doesn't answer three real production questions: where the configuration comes from, how "running" is told apart from "healthy," and what happens if traffic changes. This module's three pieces — ConfigMap/Secret, probes, HorizontalPodAutoscaler — exist exactly to answer those three questions, in that order, on the same andes-cargo-status-api Deployment you already know.

Before moving on you should be able to: name the three questions this module resolves, and confirm your cluster is still in the exact same state Module 2 left it in.

Next lesson: ConfigMap, separating configuration from the image. That's where the first of the three pieces begins — why the LocalStack endpoint never should have lived, and never will live, inside the Dockerfile.

Resources

Kubernetes — Configuration Best Practices — the official guide on why to separate configuration from the image, the thread that opens this module.
aws-serverless-and-containers-guide (NIEVA), Module 6, lesson 5 — the original app.py that reads DYNAMODB_ENDPOINT_URL/AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY/SHIPMENTS_TABLE_NAME from environment variables, with no change in this module.
kubernetes-and-eks-in-production-guide (NIEVA), Module 2, lesson 8 — the exact final checklist this module starts from.

← Previous8. Project: `andes-cargo-status-api` with N replicas Next →2. ConfigMaps: separating configuration from the image