Module 2: Pods Deployments And Services

8. Project: `andes-cargo-status-api` with N replicas

Next →

Description

This project brings together, into a single system verified end to end, the three primitives you built in this module: you scale andes-cargo-status-api's Deployment to three replicas, confirm all three respond through the same status-api-service — the load balancing lesson 6 explained in theory, now with real evidence — and repeat, one last time, this module's central experiment: deleting a Pod on purpose and watching it get replaced on its own, this time with the complete system working together. Everything that follows really ran against andes-cargo-cluster.

Connection to the module

Each lesson in this module built one piece: the Pod concept (lessons 2 and 4), the Deployment/ReplicaSet concept (lessons 3 and 5), the Service concept (lessons 6 and 7). This project is the first time you see all three working together, under a condition no individual lesson tested yet: more than two replicas, with real traffic spread across all of them.

Step 1 — Scale the Deployment to 3 replicas

Modify deployment.yaml (the same file from lesson 5), changing replicas: 2 to replicas: 3:

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: andes-cargo-status-api
  namespace: andes-cargo
  labels:
    app: andes-cargo-status-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: andes-cargo-status-api
  template:
    metadata:
      labels:
        app: andes-cargo-status-api
    spec:
      containers:
        - name: andes-cargo-status-api
          image: andes-cargo-status-api:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
kubectl apply -f deployment.yaml

What to expect (configured, not created — the Deployment already existed from lesson 5, this is a field update, not a creation):

deployment.apps/andes-cargo-status-api configured

This single change — one number, from 2 to 3 — is all it took. You didn't recreate the Service, didn't touch the namespace, didn't rebuild the image. Lesson 3's reconciliation loop does the rest on its own.

Step 2 — Verify: three replicas, spread across the available nodes

kubectl get deployments -n andes-cargo

What to expect (AGE is your variable value):

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
andes-cargo-status-api   3/3     3            3           119s
kubectl get pods -n andes-cargo -o wide

What to expect (hash suffixes, IPs, and AGE are your variable values — the name prefix and the two-node pattern are literal for this cluster architecture):

NAME                                      READY   STATUS    RESTARTS   AGE    IP           NODE                          NOMINATED NODE   READINESS GATES
andes-cargo-status-api-56856576d4-7ztk9   1/1     Running   0          106s   10.244.1.3   andes-cargo-cluster-worker2   <none>           <none>
andes-cargo-status-api-56856576d4-fqn9j   1/1     Running   0          6s     10.244.2.5   andes-cargo-cluster-worker    <none>           <none>
andes-cargo-status-api-56856576d4-vjc9k   1/1     Running   0          119s   10.244.2.4   andes-cargo-cluster-worker    <none>           <none>

Three Pods, spread between andes-cargo-cluster-worker and andes-cargo-cluster-worker2 — the control-plane doesn't receive any, exactly what you predicted in lesson 5's Exercise 2. With only two worker nodes available for three replicas, one of the two necessarily ends up with two Pods — in this case, andes-cargo-cluster-worker.

Step 3 — Confirm the load balancing: all three replicas respond

Here's this project's central verification: does the Service really spread traffic across the three replicas, not just two, or always toward the same one? kubectl port-forward (lesson 7) isn't useful for this — it keeps the connection fixed to a single Pod. Instead, launch a temporary Pod inside the cluster, and from there make several requests to the Service by its internal DNS name:

kubectl run curl-client --image=curlimages/curl:latest --restart=Never -n andes-cargo -- sleep 3600

What to expect:

pod/curl-client created

Wait a few seconds for it to start, and confirm:

kubectl get pod curl-client -n andes-cargo

What to expect:

NAME          READY   STATUS    RESTARTS   AGE
curl-client   1/1     Running   0          5s

Now, from inside that Pod, make nine consecutive requests against the Service's full DNS name — the same <service>.<namespace>.svc.cluster.local pattern lesson 6 explained:

kubectl exec curl-client -n andes-cargo -- sh -c 'for i in $(seq 1 9); do curl -s http://status-api-service.andes-cargo.svc.cluster.local/health; echo; done'

What to expect (literal, executed — all nine responses are identical in content, because /health doesn't depend on which specific Pod responds, but that doesn't mean the same Pod handled all nine):

{"service":"andes-cargo-status-api","status":"ok"}
{"service":"andes-cargo-status-api","status":"ok"}
{"service":"andes-cargo-status-api","status":"ok"}
{"service":"andes-cargo-status-api","status":"ok"}
{"service":"andes-cargo-status-api","status":"ok"}
{"service":"andes-cargo-status-api","status":"ok"}
{"service":"andes-cargo-status-api","status":"ok"}
{"service":"andes-cargo-status-api","status":"ok"}
{"service":"andes-cargo-status-api","status":"ok"}

Nine 200 responses, all with the same body — as expected. The real question is which Pod handled each one, and the answer isn't in the curl, it's in each Pod's logs. Count how many GET /health lines each one has:

for p in $(kubectl get pods -n andes-cargo -l app=andes-cargo-status-api -o jsonpath='{.items[*].metadata.name}'); do
  echo "--- $p ---"
  kubectl logs "$p" -n andes-cargo | grep -c "GET /health"
done

What to expect (Pod names are your variable value; the exact counts also vary depending on how many times you've repeated the curl — kube-proxy's distribution mechanism doesn't guarantee a perfectly even split over a few requests — but the underlying pattern — no Pod at zero — is what confirms the load balancing):

--- andes-cargo-status-api-56856576d4-7ztk9 ---
3
--- andes-cargo-status-api-56856576d4-fqn9j ---
4
--- andes-cargo-status-api-56856576d4-vjc9k ---
3

There's the confirmation: all three Pods received requests (3, 4, 3 — adding up to the nine curls), not just one or two. kube-proxy, the component lesson 6 already named, spread the requests across the three IPs in the Endpoints list, with no one explicitly telling it how — the same mechanism from lesson 6, now confirmed with three replicas instead of two.

Step 4 — Delete a Pod on purpose, one last time

Repeat, for the last time in this module, the central experiment: deleting a Pod and observing the replacement — but now with three replicas and real traffic flowing through the Service.

kubectl get pods -n andes-cargo -l app=andes-cargo-status-api -o wide

What to expect (your name/IP/AGE values are going to differ):

NAME                                      READY   STATUS    RESTARTS   AGE     IP           NODE                          NOMINATED NODE   READINESS GATES
andes-cargo-status-api-56856576d4-7ztk9   1/1     Running   0          2m16s   10.244.1.3   andes-cargo-cluster-worker2   <none>           <none>
andes-cargo-status-api-56856576d4-fqn9j   1/1     Running   0          36s     10.244.2.5   andes-cargo-cluster-worker    <none>           <none>
andes-cargo-status-api-56856576d4-vjc9k   1/1     Running   0          2m29s   10.244.2.4   andes-cargo-cluster-worker    <none>           <none>
kubectl delete pod andes-cargo-status-api-56856576d4-fqn9j -n andes-cargo

Substitute the exact name of one of your own Pods.

What to expect:

pod "andes-cargo-status-api-56856576d4-fqn9j" deleted from andes-cargo namespace
kubectl get pods -n andes-cargo -l app=andes-cargo-status-api -o wide

What to expect (andes-cargo-status-api-56856576d4-2slnk is a new Pod, with a new IP; the other two, unchanged):

NAME                                      READY   STATUS    RESTARTS   AGE     IP           NODE                          NOMINATED NODE   READINESS GATES
andes-cargo-status-api-56856576d4-2slnk   1/1     Running   0          36s     10.244.2.6   andes-cargo-cluster-worker    <none>           <none>
andes-cargo-status-api-56856576d4-7ztk9   1/1     Running   0          2m52s   10.244.1.3   andes-cargo-cluster-worker2   <none>           <none>
andes-cargo-status-api-56856576d4-vjc9k   1/1     Running   0          3m5s    10.244.2.4   andes-cargo-cluster-worker    <none>           <none>
kubectl get deployment andes-cargo-status-api -n andes-cargo

What to expect (never dropped from 3/3):

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
andes-cargo-status-api   3/3     3            3           3m5s

Step 5 — Confirm the whole system stays healthy after the replacement

One last check: with the new Pod already running, confirm the Service automatically included it in its Endpoints list — with no one asking — and that it's still balancing correctly:

kubectl exec curl-client -n andes-cargo -- sh -c 'for i in $(seq 1 6); do curl -s -o /dev/null -w "%{http_code}\n" http://status-api-service.andes-cargo.svc.cluster.local/health; done'

What to expect:

200
200
200
200
200
200
for p in $(kubectl get pods -n andes-cargo -l app=andes-cargo-status-api -o jsonpath='{.items[*].metadata.name}'); do
  echo "--- $p ---"
  kubectl logs "$p" -n andes-cargo | grep -c "GET /health"
done

What to expect (the counts are accumulated since each Pod started — the replacement Pod, younger, may show a different number from the other two; what matters is that none stays at zero):

--- andes-cargo-status-api-56856576d4-2slnk ---
3
--- andes-cargo-status-api-56856576d4-7ztk9 ---
3
--- andes-cargo-status-api-56856576d4-vjc9k ---
6

The new Pod (2slnk) is already receiving traffic, with nothing special you had to configure — the Service's selector found it the moment it was born, exactly the automatic mechanism lesson 6 described.

Clean up the temporary test Pod, you no longer need it:

kubectl delete pod curl-client -n andes-cargo

What to expect:

pod "curl-client" deleted from andes-cargo namespace

The final checklist: this module's complete system

kubectl get all -n andes-cargo

What to expect (Pod names and IPs are variable; the general shape — one Deployment, one ReplicaSet, three Pods, one Service — is literal):

NAME                                          READY   STATUS    RESTARTS   AGE
pod/andes-cargo-status-api-56856576d4-2slnk   1/1     Running   0          48s
pod/andes-cargo-status-api-56856576d4-7ztk9   1/1     Running   0          3m4s
pod/andes-cargo-status-api-56856576d4-vjc9k   1/1     Running   0          3m17s

NAME                         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/status-api-service   ClusterIP   10.96.78.1   <none>        80/TCP    2m1s

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/andes-cargo-status-api   3/3     3            3           3m17s

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/andes-cargo-status-api-56856576d4   3         3         3       3m17s

Four object types, one single hierarchy: the Deployment you declared (lesson 5, scaled in this project's Step 1), the ReplicaSet it created underneath (lesson 3, never touched directly), the three Pods it keeps alive (confirmed healthy and balanced in this project), and the Service that gives them a stable address (lesson 7). No object in this system depends on you remembering any specific Pod's IP or name — that is, on a single screen, this module's whole thesis.

Common mistakes

Verifying load balancing with curl straight from your machine instead of from inside the cluster, and seeing nothing distributed (workflow, specific to this project). What happens: someone tries to repeat Step 3's verification using kubectl port-forward (lesson 7's pattern) in a loop, and notices the same Pod handles every request, no matter how many times they repeat the curl. Why it happens: kubectl port-forward keeps a fixed connection to a specific Pod for the tunnel's whole life — it doesn't split individual requests across the available Pods, unlike a real request made directly against the Service's IP from inside the cluster. How to spot it: if you see the same Pod responding to GET /health in its logs, over and over, while other healthy Pods stay at zero. How to fix it: to verify real load balancing between replicas, the request has to reach the Service's IP (or its DNS name) directly from inside the cluster — this project's exact pattern, with a temporary Pod (curl-client) making the requests — not through a port-forward tunnel, which fixes the connection to a single backend.

Expecting a perfectly even distribution (3/3/3) in just nine requests, and suspecting a problem if you don't see it (conceptual). What happens: someone runs Step 3's verification, sees a split like 3/4/2 instead of 3/3/3, and assumes something's wrong with the Service or with kube-proxy. Why it happens: "load balancing" sounds like "mathematically exact split," but kube-proxy's real mechanism (based on iptables rules by default) doesn't guarantee a perfect distribution in a small sample — it does guarantee, with enough traffic volume, that no healthy Pod is permanently left without receiving requests. How to spot it: if your own nine-request split didn't come out exactly 3/3/3. How to fix it: what matters to verify isn't the exact proportion, but that no healthy Pod stays at zero — that's what confirms the Service is considering all three, not that the split is mathematically perfect in such a small sample.

Forgetting to delete curl-client at the end, and confusing it in a future lesson or module for a real Andes Cargo object (discipline). What happens: someone finishes this project without running Step 5's kubectl delete pod curl-client, and in a later module (for example, Module 6, when Gatekeeper starts requiring resource limits across the whole andes-cargo namespace) runs into a Pod with no relation to andes-cargo-status-api violating a policy, without remembering where it came from. Why it happens: it's easy to forget a temporary debugging Pod once it's served its immediate purpose. How to spot it: kubectl get pods -n andes-cargo shows a Pod named curl-client, with no app=andes-cargo-status-api label, long after this project ended. How to fix it: this project's Step 5 includes explicit cleanup for this exact reason — any temporary debugging Pod you create for the rest of this guide (you're going to repeat this pattern) should be deleted as soon as it's served its purpose, not left running indefinitely inside andes-cargo.

Exercises

Exercise 1 — Reconstruct the whole project from memory. Without going back to the lesson, list this project's five steps, in order, and what each one confirmed.

See solution
  1. Scale deployment.yaml from replicas: 2 to replicas: 3, and apply it — confirms a single number changes everything needed.
  2. Verify with kubectl get deployments/kubectl get pods -o wide that all three replicas are healthy, spread across the two worker nodes.
  3. Launch a temporary Pod (curl-client) inside the cluster, and make several requests against the Service's DNS name — confirms, with each Pod's logs, that all three replicas receive traffic, not just one or two.
  4. Delete a Pod on purpose, and confirm the Deployment never drops below 3/3 — the ReplicaSet replaces the lost replica immediately.
  5. Re-verify the load balancing after the replacement, confirming the new Pod automatically joins the Service's Endpoints list, with no additional configuration.

Exercise 2 — Explain why port-forward doesn't work for verifying load balancing. Without going back to "Common mistakes," explain in two or three sentences why kubectl port-forward isn't the right tool to confirm a Service spreads traffic across several replicas, and which tool is.

See solution

kubectl port-forward opens a fixed tunnel to a single backend (a specific Pod, chosen once when the tunnel starts) and keeps that same connection for every request that goes through it during its lifetime — it doesn't re-check the Service's Endpoints list for each individual request. To see real load balancing, the request has to originate inside the cluster and reach the Service's IP (or DNS name) directly, letting kube-proxy decide, request by request, which Pod to route it to — the pattern this project used with the temporary curl-client Pod.

Exercise 3 — Design a verification for five replicas. If this same Deployment had replicas: 5 instead of 3, and you wanted to confirm all five receive traffic, how many minimum requests would you run in Step 3's loop to have a reasonable probability that no healthy replica stays at zero, and why more than five?

See solution

Running exactly five requests wouldn't be enough for reasonable confidence — since kube-proxy's split doesn't guarantee a perfectly uniform distribution (confirmed in this lesson's "Common mistakes"), with a sample the same size as the replica count there's a real chance that, by simple randomness in the distribution sequence, some replica ends up at zero purely by statistical bad luck, not any real problem. A reasonable rule of thumb is to run a clear multiple of the replica count — for example, fifteen or twenty requests for five replicas (three or four times the number of backends) — so the probability of a healthy replica staying at zero from simple statistical variation is low, without needing an exaggeratedly large number.

Summary and next step

This project closed Module 2 with the complete system working together: you scaled andes-cargo-status-api to three replicas with a single number change, confirmed with real evidence — per-Pod log counts, not just the word "balancing" — that all three receive traffic through status-api-service, and repeated, one last time, this module's central experiment: deleting a Pod on purpose, watching the ReplicaSet replace it immediately while the Service kept responding with no visible interruption. The final checklist (kubectl get all -n andes-cargo) leaves you with this module's four pieces — Deployment, ReplicaSet, three Pods, one Service — on a single screen.

Before moving on you should be able to: scale an existing Deployment without recreating any other object; verify real load balancing between replicas using a temporary Pod inside the cluster, distinguishing that technique from port-forward; and reconstruct, from memory, why deleting a managed Pod never reduces the number of available replicas for more than an instant.

Next module: configuration, secrets, health, and autoscaling. Module 3 takes exactly this same Deployment — healthy, with three replicas, balanced — and resolves what lesson 7 of this module honestly left pending: ConfigMap and Secret so andes-cargo-status-api has the configuration it's missing, liveness/readiness/startup probes so Kubernetes knows how to tell a healthy Pod apart from one that only looks healthy, and a HorizontalPodAutoscaler that scales replicas based on real metrics, not a fixed number you decide by hand each time.

Resources

  1. Kubernetes — Scaling a Deployment — the official guide to scaling a Deployment, this project's Step 1 central operation.
  2. Kubernetes — Debug Services — official reference for Service/Endpoints diagnostics, the technical foundation of this project's load-balancing verification.
  3. Kubernetes — kube-proxy — official reference for the component responsible for spreading traffic across replicas, mentioned in this lesson and developed in depth in Module 4.
  4. aws-serverless-and-containers-guide (NIEVA), Module 7, lesson 8 — the documented-ECS-only status-api-service this module finished replacing with a real Kubernetes Service, running with balanced replicas.
← Previous7. Hands-on: exposing `status-api-service`Next →1. Module introduction: what a Deployment alone doesn't give you