Module 3: Configuration Secrets Health And Autoscaling

7. `HorizontalPodAutoscaler`: scaling by metrics, not by gut feeling

Next →

Description

Every time this guide changed andes-cargo-status-api's replica count — from 2 to 3, in Module 2's project — you did it yourself, editing a number by hand in deployment.yaml. That works in a lab, but it's lesson 1's third unanswered question: what happens if real traffic changes faster than any person can react? This lesson installs the piece that answers that — the HorizontalPodAutoscaler — and, before getting there, resolves a real requirement almost no one anticipates the first time they need it: Kubernetes doesn't know how much CPU a Pod uses unless something tells it.

Connection to the module

This is the third and last of the three pieces lesson 1 promised. Lesson 8 — this module's project — generates real load against andes-cargo-status-api and watches this lesson's HorizontalPodAutoscaler really react, raising and lowering replicas with no one editing any YAML during the process.

The problem a HorizontalPodAutoscaler solves

A HorizontalPodAutoscaler (HPA) is a Kubernetes object that continuously and automatically adjusts a Deployment's replicas field — the same field you edited by hand in Module 2 — based on a real metric you declare. Instead of "always 3 replicas, decided once," the HPA implements "as many replicas as needed to keep average CPU usage close to a target, not one more, not one less, adjusted every few seconds."

                    WHAT A HorizontalPodAutoscaler DOES

  ┌─────────────────┐    reads metrics every  ┌──────────────────┐
  │  metrics-server   │ ◀───few seconds────────│  HorizontalPod-     │
  │  (each Pod's        │                        │  Autoscaler          │
  │   real CPU)           │                        └─────────┬──────────┘
  └─────────────────┘                                        │
                                                                │ adjusts replicas
                                                                ▼
                                                      ┌──────────────────┐
                                                      │  Deployment         │
                                                      │  andes-cargo-       │
                                                      │  status-api          │
                                                      └──────────────────┘

Analogy: the air conditioner, not the clock

A building could have a fixed rule: "the AC goes to max from 9 to 5, and shuts off the rest of the time" — a decision made once, with no relation to the real temperature at any given moment. A real thermostat does something different: it measures the real temperature, constantly, and adjusts cooling based on what it finds — stronger on an extremely hot day even if it's a holiday, barely on during a mild day even at peak hours. A Deployment with a fixed replicas: 3 is the first building: a decision made once, with no relation to real load. A HorizontalPodAutoscaler is the thermostat: it measures a real metric (CPU, in this lesson) and adjusts replicas accordingly, with no one having to watch a clock or guess when traffic is going to spike.

The requirement almost no one anticipates: metrics-server

An HPA needs to read each Pod's real CPU usage — and that data doesn't exist on a freshly installed Kubernetes cluster. kind, like most Kubernetes distributions, doesn't include metrics-server by default. Confirm it yourself before installing anything:

kubectl top nodes

What to expect (literal, on a cluster with no metrics-server):

error: Metrics API not available

kubectl top — the command that summarizes CPU/memory for nodes or Pods — depends on the same API an HPA uses underneath (metrics.k8s.io). With no metrics-server running, neither you nor the HPA have any real number to query.

Install metrics-server

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

What to expect (literal, executed — the complete list of objects the official manifest creates):

serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

Confirm the installed version:

kubectl get deployment metrics-server -n kube-system -o jsonpath='{.spec.template.spec.containers[0].image}'

What to expect (the most recent stable version at the time this guide was written — check yours, it may have moved forward):

registry.k8s.io/metrics-server/metrics-server:v0.9.0

The gotcha: kind's self-signed certificates

Wait a few seconds and try again:

kubectl top nodes

What to expect (still fails — this is the gotcha, not a mistake on your part):

error: Metrics API not available

Check metrics-server's own logs to understand why:

kubectl logs -n kube-system -l k8s-app=metrics-server --tail=20

What to expect (literal, executed):

E0814 20:07:05.393308       1 scraper.go:149] "Failed to scrape node" err="Get \"https://172.19.0.4:10250/metrics/resource\": tls: failed to verify certificate: x509: cannot validate certificate for 172.19.0.4 because it doesn't contain any IP SANs" node="andes-cargo-cluster-worker"
E0814 20:07:05.398048       1 scraper.go:149] "Failed to scrape node" err="Get \"https://172.19.0.2:10250/metrics/resource\": tls: failed to verify certificate: x509: cannot validate certificate for 172.19.0.2 because it doesn't contain any IP SANs" node="andes-cargo-cluster-control-plane"
E0814 20:07:05.398262       1 scraper.go:149] "Failed to scrape node" err="Get \"https://172.19.0.3:10250/metrics/resource\": tls: failed to verify certificate: x509: cannot validate certificate for 172.19.0.3 because it doesn't contain any IP SANs" node="andes-cargo-cluster-worker2"
I0814 20:07:25.113290       1 server.go:192] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"

The exact cause, no guessing: metrics-server needs to connect to each node's kubelet over HTTPS (port 10250) to read its metrics — and verify that TLS certificate against a certificate authority, as it does by default against a real cloud cluster. The problem is kind generates self-signed kubelet certificates, with no IP SAN (Subject Alternative Name) entry explicitly declaring the cluster's internal IPs — a normal detail for a lab cluster, which metrics-server correctly interprets as "I can't trust this certificate." This isn't a mistake in your installation: it's documented, known friction between metrics-server and any cluster with kubelet certificates unverifiable this way — including kind, minikube, and several other local distributions.

The official solution is the --kubelet-insecure-tls flag, which tells metrics-server: "connect anyway, without verifying the kubelet's certificate." It's acceptable in a local lab like this one, where you already trust the cluster's entire network; on real EKS, AWS's managed control plane resolves this problem at its root with verifiable certificates by default, and this flag isn't needed.

kubectl patch deployment metrics-server -n kube-system --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'

What to expect:

deployment.apps/metrics-server patched

Confirm the flag got added at the end of the existing argument list:

kubectl get deployment metrics-server -n kube-system -o jsonpath='{.spec.template.spec.containers[0].args}'

What to expect (literal):

["--cert-dir=/tmp","--secure-port=10250","--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname","--kubelet-use-node-status-port","--metric-resolution=15s","--kubelet-insecure-tls"]

Wait for the new metrics-server Pod to start and confirm:

kubectl top nodes

What to expect (literal, executed — CPU/memory numbers are going to vary based on your machine's real load at the exact moment you run the command):

NAME                                CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
andes-cargo-cluster-control-plane   131m         1%       974Mi           12%
andes-cargo-cluster-worker          62m          0%       403Mi           5%
andes-cargo-cluster-worker2         28m          0%       345Mi           4%
kubectl top pods -n andes-cargo

What to expect (literal — your three Pods, with real CPU, at rest):

NAME                                      CPU(cores)   MEMORY(bytes)
andes-cargo-status-api-65fcd6f6c8-45hq5   2m           40Mi
andes-cargo-status-api-65fcd6f6c8-9hbfr   1m           40Mi
andes-cargo-status-api-65fcd6f6c8-kr7hv   1m           40Mi

metrics-server is healthy, and for the first time in this guide, there's a real per-Pod CPU usage number an HPA can reference.

An additional requirement: resources.requests

An HPA that scales by CPU utilization percentage (averageUtilization) needs a reference point to calculate that percentage against — and that reference point is resources.requests.cpu, the same field that declares how much CPU each Pod "reserves" when scheduled onto a node. Without that field, "50% utilization" has no denominator to calculate against. Add it to the Deployment:

# deployment.yaml (new fragment inside containers[0])
resources:
  requests:
    cpu: "100m"
    memory: "64Mi"
  limits:
    cpu: "250m"
    memory: "128Mi"

100m means one hundred millicores, or 10% of a CPU core — the value the HPA is going to use as the "100%" reference for this specific Pod. Apply the complete Deployment with this block added:

kubectl apply -f deployment.yaml
kubectl rollout status deployment/andes-cargo-status-api -n andes-cargo --timeout=60s

What to expect (the same RollingUpdate pattern already known from lessons 4 and 6):

deployment.apps/andes-cargo-status-api configured
Waiting for deployment "andes-cargo-status-api" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 1 old replicas are pending termination...
deployment "andes-cargo-status-api" successfully rolled out

Create the HorizontalPodAutoscaler

# hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: andes-cargo-status-api-hpa
  namespace: andes-cargo
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: andes-cargo-status-api
  minReplicas: 2
  maxReplicas: 6
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 60

Every field, explained:

  • scaleTargetRef — which Deployment this HPA controls. The same by-name reference mechanism you already know from selector/matchLabels, but pointing at an entire object, not a set of Pods.
  • minReplicas/maxReplicas — the range the HPA can move within. It's never going to drop below 2 (so the Service never ends up with a single replica), nor rise above 6 (a safety ceiling, so a traffic spike doesn't consume the entire lab cluster's capacity with no limit).
  • metrics[0] — the target metric: CPU utilization, with a target of 50% of the value declared in resources.requests.cpu. If the real average rises above 50%, the HPA adds replicas; if it drops, it removes them — always within the minReplicas/maxReplicas range.
  • behavior.scaleDown.stabilizationWindowSeconds: 60 — a deliberate safeguard: when CPU drops, the HPA waits 60 seconds of consistently low metrics before reducing replicas, so it doesn't react to a momentary dip and scale back up seconds later (a pattern known as flapping). There's no equivalent stabilizationWindowSeconds for scaling up in this configuration — by design: reacting fast to a real traffic spike matters more than avoiding one extra replica for a few seconds.
kubectl apply -f hpa.yaml

What to expect:

horizontalpodautoscaler.autoscaling/andes-cargo-status-api-hpa created

Wait a few seconds — the HPA needs at least one cycle of its sync period (15 seconds, by default) to calculate the current metric — and confirm:

kubectl get hpa -n andes-cargo

What to expect (literal, executed — 1% is your CPU at rest, the real value is going to vary):

NAME                         REFERENCE                           TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
andes-cargo-status-api-hpa   Deployment/andes-cargo-status-api   cpu: 1%/50%   2         6         3          33s

cpu: 1%/50% — well below the target, so the HPA has no reason to scale yet. REPLICAS: 3 because the Deployment already had 3 replicas when the HPA started controlling it, a number within the 2-6 range the HPA has no reason to touch at rest.

Confirm the complete behavior policy:

kubectl describe hpa andes-cargo-status-api-hpa -n andes-cargo

What to expect (relevant fragment, literal):

Behavior:
  Scale Up:
    Stabilization Window: 0 seconds
    Select Policy: Max
    Policies:
      - Type: Pods     Value: 4    Period: 15 seconds
      - Type: Percent  Value: 100  Period: 15 seconds
  Scale Down:
    Stabilization Window: 60 seconds
    Select Policy: Max
    Policies:
      - Type: Percent  Value: 100  Period: 15 seconds

Notice Scale Up: even though hpa.yaml only explicitly declared behavior.scaleDown, Kubernetes automatically filled in a default Scale Up policy — aggressive on purpose, to react quickly to a real spike — while Scale Down uses exactly the 60 seconds you did declare. This asymmetry — up fast, down cautiously — is the same philosophy you already saw in lesson 5's probes: the "cheap" consequence (adding capacity) fires more freely than the consequence that's "expensive to get wrong" (suddenly removing capacity).

Common mistakes

Creating the HPA before the Deployment finishes its RollingUpdate with resources.requests included, and seeing a transient metrics error. What happens: if the HPA is created while Pods from the old template (with no resources.requests) still exist, the HPA can briefly report an error like failed to get cpu utilization: missing request for cpu in container ... of Pod .... Why it happens: an HPA calculating utilization percentage needs every Pod it's averaging to have resources.requests.cpu declared — a single Pod without that field invalidates the entire calculation. How to spot it: kubectl describe hpa shows a Warning FailedGetResourceMetric event mentioning a specific Pod. How to fix it: wait for kubectl rollout status to confirm the RollingUpdate finished completely before creating the HPA — the error resolves itself the moment every active Pod shares the same template with resources.requests.

Forgetting --kubelet-insecure-tls and assuming metrics-server is broken (expectation, this lesson's central gotcha). What happens: someone installs metrics-server with the official manifest, sees error: Metrics API not available persisting after waiting, and concludes the project has a bug. How to spot it: kubectl logs -n kube-system -l k8s-app=metrics-server specifically shows x509: cannot validate certificate ... because it doesn't contain any IP SANs. How to fix it: this lesson documented the exact cause and the fix — --kubelet-insecure-tls — because it's known, expected friction on kind (and on most local clusters), not a flaw in the metrics-server project or your installation.

Confusing resources.requests with resources.limits when thinking about the HPA's target (conceptual). What happens: someone assumes averageUtilization: 50 is calculated against resources.limits.cpu (250m in this lesson), not against resources.requests.cpu (100m), and is surprised the HPA scales "earlier than expected." How to spot it: if your manual calculations of "at what real CPU percentage should it scale" don't match what you observe. How to fix it: a Utilization-type HPA always calculates the percentage against requests, never against limits — with requests.cpu: 100m and a 50% target, the HPA reacts when real usage crosses 50m per Pod, no matter how far that is from limits.cpu: 250m.

Exercises

Exercise 1 — Reconstruct the complete installation sequence. Without going back to the lesson, list the steps, in order, from confirming metrics-server doesn't exist to seeing the first kubectl get hpa with a real number.

See solution
  1. Confirm with kubectl top nodes that there's no metrics-server (error: Metrics API not available).
  2. Install the official manifest with kubectl apply -f .../components.yaml.
  3. Confirm kubectl top nodes still fails, and diagnose the cause with kubectl logs (self-signed certificates with no IP SANs).
  4. Patch metrics-server's Deployment to add --kubelet-insecure-tls.
  5. Confirm kubectl top nodes/kubectl top pods working.
  6. Add resources.requests/resources.limits to andes-cargo-status-api's Deployment, apply, and wait for the RollingUpdate.
  7. Create hpa.yaml, apply it, and confirm kubectl get hpa showing a real CPU percentage.

Exercise 2 — Explain the TLS gotcha to a colleague without using the word "certificate." In two or three sentences, without using the word "certificate" or "TLS," explain why metrics-server fails on kind without --kubelet-insecure-tls.

See solution

A reasonable answer: "metrics-server needs to trust each node's identity before reading its metrics, and on a lab cluster like kind, that identity isn't signed in a way metrics-server can automatically verify by default. The flag explicitly tells it 'trust it anyway' — reasonable on a local lab, but something a real cloud cluster like EKS doesn't need, because there each node's identity is verifiable in a standard way."

Exercise 3 — Calculate when this HPA would scale. With resources.requests.cpu: 100m and averageUtilization: 50, starting at how many millicores of average usage per Pod would this lesson's HPA start adding replicas?

See solution

Starting at 50m of average usage per Pod (50% of the 100m declared in requests.cpu). If the current Pods' real average sustainedly exceeds that threshold, the HPA calculates how many additional replicas would be needed to bring the average back down close to the target, and adjusts replicas accordingly — the exact mechanism lesson 8 is going to trigger with real load.

Summary and next step

This lesson installed this module's third and final piece: metrics-server, with kind's real self-signed-certificate gotcha documented and resolved (--kubelet-insecure-tls); resources.requests/resources.limits on the Deployment, the reference point any utilization-based HPA needs; and the andes-cargo-status-api-hpa HorizontalPodAutoscaler itself, with a 2 to 6 replica range and a 50% CPU target. At rest, the HPA has no reason to act — you confirmed it with cpu: 1%/50% — but the complete mechanism is already in place, waiting for a real condition.

Before moving on you should be able to: explain why metrics-server doesn't come included by default on kind, and diagnose the certificate error with no guessing; explain why resources.requests (not limits) is a Utilization-type HPA's denominator; and read a complete kubectl describe hpa, including its Scale Up/Scale Down policies.

Next lesson: this module's project, andes-cargo-status-api under load. There you generate real traffic, watch the HPA really react — replicas rising when CPU rises, dropping when the load stops — and close the whole module with the entire system working together.

Resources

  1. Kubernetes — Horizontal Pod Autoscaling — the complete official guide for this lesson's mechanism.
  2. Kubernetes — HorizontalPodAutoscaler Walkthrough — official step-by-step tutorial, with the same resources.requests + HPA pattern from this lesson.
  3. kubernetes-sigs/metrics-server — the project's official repository, including --kubelet-insecure-tls's documentation and why it's needed on local clusters.
  4. kind — Known Issues — kind's official documentation on known local-cluster friction, the same kind of gotcha (self-signed kubelet certificates) this lesson documented step by step.
← Previous6. Hands-on: real probes on `status-api-service`Next →8. Project: `andes-cargo-status-api` under load