Module 3: Configuration Secrets Health And Autoscaling

8. Project: `andes-cargo-status-api` under load

Next →

Description

This project closes Module 3 with the test all the previous lessons prepared for: you're going to generate real traffic against andes-cargo-status-api, watch lesson 7's HorizontalPodAutoscaler really react — replicas rising while load rises — and then, after stopping the load, watch those same replicas drop on their own, with no one editing deployment.yaml even once. Everything that follows ran against andes-cargo-cluster, with the exact replica count at each point being your own variable value — the mechanism, not the number, is what this project demonstrates.

Connection to the module

Each lesson in this module built one piece: ConfigMap/Secret (lessons 2-4), probes (lessons 5-6), HorizontalPodAutoscaler/metrics-server (lesson 7). This project is the first time you see the third piece working under a real condition — not at rest, like lesson 7 ended — closing the complete arc lesson 1 opened: from "3 replicas running" to a system with externalized configuration, health verification, and capacity that adjusts itself.

Step 1 — Confirm the baseline before generating load

kubectl get hpa -n andes-cargo

What to expect (your resting CPU percentage may vary slightly, but should be well below 50%):

NAME                         REFERENCE                           TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
andes-cargo-status-api-hpa   Deployment/andes-cargo-status-api   cpu: 1%/50%   2         6         3          33s
kubectl top pods -n andes-cargo -l app=andes-cargo-status-api

What to expect (real CPU, very low, at rest):

NAME                                      CPU(cores)   MEMORY(bytes)
andes-cargo-status-api-548966dd97-cwtlv   2m           40Mi
andes-cargo-status-api-548966dd97-ksl7m   2m           40Mi
andes-cargo-status-api-548966dd97-qbl4h   1m           40Mi

This is your "before" snapshot — keep it in mind to compare against what follows.

Step 2 — The load generator

A single sequential HTTP client doesn't generate enough work to move the CPU needle for a service as lightweight as /health — you need real concurrency. This load generator launches 30 parallel loops inside a single busybox Pod, each one requesting /health nonstop:

# load-generator.yaml
apiVersion: v1
kind: Pod
metadata:
  name: load-generator
  namespace: andes-cargo
  labels:
    app: load-generator
spec:
  restartPolicy: Never
  containers:
    - name: load-generator
      image: busybox:1.36
      command: ["/bin/sh", "-c"]
      args:
        - |
          for i in $(seq 1 30); do
            (while true; do wget -q -O- http://status-api-service.andes-cargo.svc.cluster.local/health >/dev/null; done) &
          done
          wait

Notice the Service's DNS name (status-api-service.andes-cargo.svc.cluster.local) — the same <service>.<namespace>.svc.cluster.local pattern you already know from Module 2, and in the same andes-cargo namespace as the service itself, so traffic stays entirely inside the cluster.

kubectl apply -f load-generator.yaml

What to expect:

pod/load-generator created

Step 3 — Watch the HPA react: replicas going up

kubectl top pods -n andes-cargo -l app=andes-cargo-status-api
kubectl get hpa -n andes-cargo

What to expect (~10 seconds after starting the load — literal, executed; your CPU numbers and the exact moment it crosses the threshold are going to vary):

NAME                                      CPU(cores)   MEMORY(bytes)
andes-cargo-status-api-548966dd97-cjf9d   130m         41Mi
andes-cargo-status-api-548966dd97-cwtlv   102m         41Mi
andes-cargo-status-api-548966dd97-cxf5t   142m         41Mi
andes-cargo-status-api-548966dd97-ksl7m   76m          40Mi
andes-cargo-status-api-548966dd97-qbl4h   97m          41Mi
andes-cargo-status-api-548966dd97-tdqqs   136m         41Mi

NAME                         REFERENCE                           TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
andes-cargo-status-api-hpa   Deployment/andes-cargo-status-api   cpu: 95%/50%   2         6         6          117s

cpu: 95%/50% — well above the target. REPLICAS: 6 — the HPA already scaled to the maximum maxReplicas allows, and you can already count six real Pods with CPU well above their 100m request each. No one edited deployment.yaml: the HPA calculated, on its own, how many replicas were needed to try to bring the average down toward the target, and adjusted replicas directly on the Deployment.

Confirm the exact event that recorded that decision:

kubectl describe hpa andes-cargo-status-api-hpa -n andes-cargo

What to expect (Events fragment, literal — Age is your variable value):

Events:
  Type    Reason              Age    From                       Message
  ----    ------              ----   ----                       -------
  Normal  SuccessfulRescale   3m52s  horizontal-pod-autoscaler  New size: 6; reason: cpu resource utilization (percentage of request) above target

New size: 6; reason: cpu resource utilization (percentage of request) above target — the HPA documents, in readable text, exactly why it made the decision it made. No ambiguity about the cause.

Step 4 — Stop the load

kubectl delete pod load-generator -n andes-cargo --wait=false

What to expect:

pod "load-generator" deleted from andes-cargo namespace

Honest note: a load generator as aggressive as 30 concurrent wget loops inside a single Pod can exhaust its own ephemeral ports before you decide to stop it — you're going to see messages like wget: can't connect to remote host: Cannot assign requested address in its logs if you leave it running for several minutes. That's not a problem with the Service or the Deployment: it's the generator itself running out of local network resources to open more connections. For this project's purpose — crossing the threshold once and watching the mechanism react — there's no need to sustain the load for more than a minute or two.

Step 5 — Watch the HPA react: replicas going down

CPU drops almost immediately once the load generator stops sending traffic — but the HPA, by design (stabilizationWindowSeconds: 60 in hpa.yaml, lesson 7), waits before reducing replicas, so it doesn't react to a momentary dip:

kubectl get hpa -n andes-cargo

What to expect (right after stopping the load — CPU already dropped, but REPLICAS hasn't yet, because the stabilization window is still running):

NAME                         REFERENCE                           TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
andes-cargo-status-api-hpa   Deployment/andes-cargo-status-api   cpu: 1%/50%   2         6         6          2m47s

Wait about a minute (the full stabilizationWindowSeconds: 60 window) and repeat:

kubectl get hpa -n andes-cargo

What to expect (literal, executed — REPLICAS dropped, straight to minReplicas, with no intermediate steps, because lesson 7's Scale Down policy allows reducing up to 100% in a single 15-second cycle once the stabilization window is satisfied):

NAME                         REFERENCE                           TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
andes-cargo-status-api-hpa   Deployment/andes-cargo-status-api   cpu: 1%/50%   2         6         2          4m47s

Confirm the second event, the exact counterpart of the first:

kubectl describe hpa andes-cargo-status-api-hpa -n andes-cargo

What to expect (Events fragment, literal):

Events:
  Type    Reason              Age    From                       Message
  ----    ------              ----   ----                       -------
  Normal  SuccessfulRescale   3m52s  horizontal-pod-autoscaler  New size: 6; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale   52s    horizontal-pod-autoscaler  New size: 2; reason: All metrics below target

Both events, one under the other: New size: 6 when CPU rose, New size: 2 when it dropped and stayed down long enough. That's the complete cycle — up and down, both automatic, both documented by the HPA itself with its exact reason — with you never touching deployment.yaml at any point in this project.

     THIS PROJECT'S COMPLETE CYCLE (numbers — yours are going to vary)

  at rest        high load             load stopped            settled
  REPLICAS: 3 ──▶ REPLICAS: 6 ────────▶ REPLICAS: 6 (waiting) ──▶ REPLICAS: 2
  cpu: 1%/50%     cpu: 95%/50%          cpu: 1%/50%               cpu: 1%/50%
                  "above target"                                  "below target"
                                        (60s stabilization
                                         window before dropping)

Step 6 — The final checklist: this module's complete system

kubectl get all -n andes-cargo
kubectl get configmap,secret -n andes-cargo
kubectl get hpa -n andes-cargo

What to expect (Pod names and the final REPLICAS are your variable value, depending on when you run this relative to the HPA's cycle; object names — Deployment, Service, ConfigMap, Secret, HorizontalPodAutoscaler — are literal):

NAME                                          READY   STATUS    RESTARTS   AGE
pod/andes-cargo-status-api-548966dd97-cwtlv   1/1     Running   0          6m17s
pod/andes-cargo-status-api-548966dd97-ksl7m   1/1     Running   0          6m5s

NAME                         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/status-api-service   ClusterIP   10.96.78.1   <none>        80/TCP    40m

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/andes-cargo-status-api   2/2     2            2           41m

NAME                                      DATA   AGE
configmap/andes-cargo-status-api-config   3      19m

NAME                                    TYPE     DATA   AGE
secret/andes-cargo-status-api-secrets   Opaque   2      19m

NAME                                                             REFERENCE                           TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/andes-cargo-status-api-hpa   Deployment/andes-cargo-status-api   cpu: 2%/50%   2         6         2          5m50s

Six object types, one single story: the Deployment Module 2 declared, now with ConfigMap/Secret mounted (lessons 2-4), probes continuously verifying its health (lessons 5-6), and a HorizontalPodAutoscaler deciding, on its own, how many replicas are needed at each moment (lessons 7-8) — the same status-api-service Service as always, with no change, serving a stable front regardless of how many replicas are behind it at any exact moment.

Common mistakes

Expecting an exact replica count at the end of the experiment, instead of the mechanism (expectation, the most important one to prevent in this project). What happens: someone compares their own REPLICAS: 2 (or whatever number) against someone else's, and worries something's wrong if it doesn't match exactly. Why it happens: the rest of this guide has a lot of perfectly literal output, and it's easy to generalize that expectation to this project too. How to spot it: if you compare your final replica count against a specific "expected" value. How to fix it: this guide's honesty rule (named since the design stage) explicitly marks a HorizontalPodAutoscaler event's final replica count as variable — it depends on the real load generated, your machine's speed, and the exact moment you ran each command. What's literal, and what this project demonstrates, is the mechanism: it rises when CPU crosses the threshold, drops when it stays below, both with a reason documented in Events.

Interpreting wget's exhausted-port messages as a Service or Deployment failure (discipline). What happens: someone leaves load-generator running for several minutes, sees error messages in its logs, and starts reviewing service.yaml or deployment.yaml looking for a problem. How to spot it: the exact message (Cannot assign requested address) is a local network error from the generator Pod itself, not an HTTP response from andes-cargo-status-api. How to fix it: as Step 4 warned, a synthetic load generator with too much concurrency can exhaust its own ephemeral ports before the real target (the Service) has any problem — the HPA already served its purpose once you crossed the threshold once, no need to sustain the load indefinitely.

Creating lesson 7's HPA more than once, without noticing (workflow). What happens: someone runs kubectl apply -f hpa.yaml again in this project, expecting it to "restart" the HPA, and instead Kubernetes simply confirms the object already exists with no changes (unchanged, not created). How to spot it: if your kubectl apply -f hpa.yaml's result says configured or unchanged instead of created. How to fix it: there's no need to recreate the HPA for this project — the same object from lesson 7 is still running, continuously evaluating the metric; you only need to generate real load to see it react, not redeclare it.

Exercises

Exercise 1 — Reconstruct the whole cycle from memory. Without going back to the project, draw or describe, in your own words, this project's complete sequence: from the baseline to the final state, including both SuccessfulRescale events.

See solution
  1. Confirm the baseline: low CPU, REPLICAS at the number inherited from Module 2 (or whatever it was when the HPA was created).
  2. Apply load-generator.yaml, generating 30 concurrent traffic loops against /health.
  3. Watch cpu rise well above 50%, and REPLICAS rise up to maxReplicas (6) — confirmed with a SuccessfulRescale ... reason: cpu resource utilization (percentage of request) above target event.
  4. Stop the load generator.
  5. Watch cpu drop almost immediately, but REPLICAS wait for the stabilizationWindowSeconds: 60 before reducing.
  6. Confirm REPLICAS dropping to minReplicas (2), with a second SuccessfulRescale ... reason: All metrics below target event.

Exercise 2 — Explain why REPLICAS dropped straight to 2, with no intermediate steps. With lesson 7's Scale Down policy in mind (Percent: 100, Period: 15 seconds), explain why this project's HPA went from 6 to 2 replicas in a single cycle, instead of dropping gradually.

See solution

The Scale Down policy allows reducing up to 100% of current replicas in a 15-second period — meaning there's no limit forcing a gradual reduction, beyond the 60-second Stabilization Window that must be satisfied before reducing at all. Once that window was satisfied, and with metrics already well below the target (cpu: 1%/50%), the HPA calculated that the correct replica count for that CPU level was the allowed minimum (minReplicas: 2), and applied that change all at once, with no artificial intermediate steps.

Exercise 3 — Design an experiment with a lower maxReplicas. If hpa.yaml had maxReplicas: 4 instead of 6, and you generated this project's same load (which in this lesson's actual run pushed utilization to 95%-183%), what replica count would you expect to see at the peak, and why?

See solution

You'd expect to see REPLICAS: 4 at the peak — maxReplicas is a hard ceiling the HPA never crosses, no matter how far above target the real metric is. Even if the HPA's math suggested more than 4 replicas were needed to bring utilization down to target, the HPA would stop at the declared maximum, and CPU utilization would stay above the 50% target until the real load dropped on its own — the reason maxReplicas should be chosen with the cluster's real available capacity in mind, not just as an arbitrary number.

Summary and next step

This project closed Module 3 with the complete system working together: you generated real load with a 30-concurrent-loop load-generator, watched the HPA really scale — from 3 to 6 replicas, with the SuccessfulRescale ... above target event as evidence — stopped the load, and watched that same HPA reduce replicas back to the minimum — SuccessfulRescale ... All metrics below target, after respecting its 60-second stabilization window. The final checklist confirmed this module's three pieces working on the same andes-cargo-status-api Deployment and status-api-service Service from Module 2: ConfigMap/Secret mounted, health probes active, and a HorizontalPodAutoscaler adjusting capacity on its own.

Before moving on you should be able to: explain why an autoscaling event's final replica count is variable, while the mechanism is literal; read an HPA's Events to confirm the exact reason for each scaling decision; and reconstruct, from memory, this project's complete up-and-down cycle.

Next module: networking, Ingress, and NetworkPolicy. Module 4 takes exactly this same system — configured, with health verified, scaling on its own — and exposes it to the cluster's external traffic with Ingress (replacing the temporary kubectl port-forward you've used since Module 2), and protects it with NetworkPolicy, closing the textual delegation cloud-security-and-guardrails-guide made to this guide.

Resources

  1. Kubernetes — Horizontal Pod Autoscaling: Autoscaling on multiple metrics and custom metrics — official reference on how an HPA can scale on more than one metric, a case this project doesn't cover but the official documentation extends directly.
  2. Kubernetes — HorizontalPodAutoscaler Walkthrough: Autoscaling Behavior — the exact official section on stabilizationWindowSeconds and the Scale Up/Scale Down policies this project watched in action.
  3. kubernetes-and-eks-in-production-guide (NIEVA), Module 2, lesson 8 — the previous project, with the same Deployment/Service, this whole module builds on.
← Previous7. `HorizontalPodAutoscaler`: scaling by metrics, not by gut feelingNext →1. Module introduction: the layer that's "not batteries included"