Module 3: Configuration Secrets Health And Autoscaling

6. Hands-on: real probes on `status-api-service`

Description

This is the lesson where andes-cargo-status-api receives its three health probes, and where you're going to induce — on purpose, in a controlled, reversible way — exactly the two failures lesson 5 described in theory: a Pod that leaves the Service without restarting, and a Pod that really restarts. None of this is simulated: you're going to edit the probe configuration to induce each failure, observe the real consequence with kubectl get pods/kubectl describe, and revert each change before continuing. Everything that follows ran against andes-cargo-cluster.

Connection to the module

This lesson closes the second of this module's three pieces with real evidence, exactly as lesson 4 closed the first. Lesson 7 adds the third — elastic capacity — on top of this same Deployment, now with configuration and health resolved.

Step 1 — Add the three probes to the `Deployment`

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: andes-cargo-status-api
  namespace: andes-cargo
  labels:
    app: andes-cargo-status-api
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: andes-cargo-status-api
  template:
    metadata:
      labels:
        app: andes-cargo-status-api
    spec:
      containers:
        - name: andes-cargo-status-api
          image: andes-cargo-status-api:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          envFrom:
            - configMapRef:
                name: andes-cargo-status-api-config
            - secretRef:
                name: andes-cargo-status-api-secrets
          startupProbe:
            httpGet:
              path: /health
              port: 8080
            periodSeconds: 5
            failureThreshold: 6
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 2
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 1
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 3

Two new things versus lesson 4's deployment.yaml: the three probes (all against /health, as lesson 5 recommended), and explicit strategy.rollingUpdate, with maxSurge: 1/maxUnavailable: 0 — this tells Kubernetes "never drop below 3 available replicas during a deployment, and add at most 1 extra Pod at a time." You're going to use this configuration on purpose in the following steps: it's going to let you induce a failure on a new Pod, with no risk to even a second of availability for the three existing Pods, which keep handling real traffic the whole time.

Apply the change, with failureThreshold: 6 (startupProbe) giving plenty of margin for Flask to start — in practice, it starts in under a second, so you're never going to see this probe fail in this guide:

kubectl apply -f deployment.yaml
kubectl rollout status deployment/andes-cargo-status-api -n andes-cargo --timeout=60s

What to expect (literal, executed):

deployment.apps/andes-cargo-status-api configured
Waiting for deployment "andes-cargo-status-api" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 1 old replicas are pending termination...
deployment "andes-cargo-status-api" successfully rolled out

Confirm the three probes are configured exactly as you declared them:

kubectl describe pod <your-pod> -n andes-cargo | grep -A3 "Liveness:\|Readiness:\|Startup:"

What to expect (literal — Kubernetes shows each probe's parameters on a single readable line):

Liveness:       http-get http://:8080/health delay=5s timeout=3s period=5s #success=1 #failure=3
Readiness:      http-get http://:8080/health delay=2s timeout=3s period=5s #success=1 #failure=1
Startup:        http-get http://:8080/health delay=0s timeout=1s period=5s #success=1 #failure=6

Also confirm all three Pods are healthy and on the Service's Endpoints list:

kubectl get endpoints status-api-service -n andes-cargo

What to expect (three IPs — yours are going to be different):

Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME                 ENDPOINTS                                         AGE
status-api-service   10.244.1.7:8080,10.244.2.8:8080,10.244.2.9:8080   23m

This is your baseline: three Pods, three Endpoints, zero RESTARTS. Everything that follows compares against this snapshot.

Step 2 — Induce a `readiness` failure

A readinessProbe can't be edited on a Pod that's already running — it's an immutable field of the container's spec — so to induce a real failure, you're going to change the probe's path on the Deployment. With maxSurge: 1/maxUnavailable: 0, this creates exactly one new Pod with the broken probe, while your three healthy Pods keep handling traffic with no interruption whatsoever:

kubectl patch deployment andes-cargo-status-api -n andes-cargo --type=json \
  -p '[{"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/readyz-check"}]'

/readyz-check doesn't exist in app.py — on purpose — so any request to that path is going to respond 404, and Kubernetes is going to count that as a probe failure.

What to expect:

deployment.apps/andes-cargo-status-api patched

Watch what happens over the next few seconds:

kubectl get pods -n andes-cargo -l app=andes-cargo-status-api -o wide

What to expect (literal, executed ~20 seconds after the patch — the three old Pods, untouched, stay 1/1 Running; the new Pod gets stuck at 0/1, Running, with RESTARTS: 0):

NAME                                      READY   STATUS    RESTARTS   AGE     IP            NODE                          NOMINATED NODE   READINESS GATES
andes-cargo-status-api-65fcd6f6c8-2bkht   1/1     Running   0          5m43s   10.244.1.7    andes-cargo-cluster-worker2   <none>           <none>
andes-cargo-status-api-65fcd6f6c8-cnbjw   1/1     Running   0          5m37s   10.244.2.9    andes-cargo-cluster-worker    <none>           <none>
andes-cargo-status-api-65fcd6f6c8-m7qf8   1/1     Running   0          5m49s   10.244.2.8    andes-cargo-cluster-worker    <none>           <none>
andes-cargo-status-api-d6cd4d46b-79hdq    0/1     Running   0          20s     10.244.2.10   andes-cargo-cluster-worker    <none>           <none>

This is exactly the behavior lesson 5 described: STATUS: Running (the process is still alive), READY: 0/1 (the readiness probe is failing), RESTARTS: 0 (no one restarted anything). Confirm the Service noticed, with no one explicitly telling it:

kubectl get endpoints status-api-service -n andes-cargo

What to expect (literal — exactly the same three IPs as before; the new Pod, 10.244.2.10, does not appear):

Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME                 ENDPOINTS                                         AGE
status-api-service   10.244.1.7:8080,10.244.2.8:8080,10.244.2.9:8080   28m

The Pod with the broken probe never made it to the Endpoints list — the Service keeps routing traffic only to the three healthy Pods, exactly as lesson 5 promised. Confirm the exact cause with kubectl describe:

kubectl describe pod andes-cargo-status-api-d6cd4d46b-79hdq -n andes-cargo

What to expect (the Events section, literal — substitute your own Pod name):

Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  29s               default-scheduler  Successfully assigned andes-cargo/andes-cargo-status-api-d6cd4d46b-79hdq to andes-cargo-cluster-worker
  Normal   Pulled     28s               kubelet            spec.containers{andes-cargo-status-api}: Container image "andes-cargo-status-api:latest" already present on machine and can be accessed by the pod
  Normal   Created    28s               kubelet            spec.containers{andes-cargo-status-api}: Container created
  Normal   Started    28s               kubelet            spec.containers{andes-cargo-status-api}: Container started
  Warning  Unhealthy  3s (x5 over 23s)  kubelet            spec.containers{andes-cargo-status-api}: Readiness probe failed: HTTP probe failed with statuscode: 404

Readiness probe failed: HTTP probe failed with statuscode: 404 — Kubernetes' exact message, with no ambiguity about the cause. Also confirm that, meanwhile, the rollout is genuinely stuck — Kubernetes is waiting, with no time limit, for this new Pod to pass its readinessProbe before it continues replacing the old Pods:

kubectl rollout status deployment/andes-cargo-status-api -n andes-cargo --timeout=5s

What to expect (the command exhausts its own 5-second timeout — the rollout itself remains stuck indefinitely until you intervene):

Waiting for deployment "andes-cargo-status-api" rollout to finish: 1 out of 3 new replicas have been updated...
error: timed out waiting for the condition

Step 3 — Fix the path, and confirm the recovery

kubectl patch deployment andes-cargo-status-api -n andes-cargo --type=json \
  -p '[{"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/health"}]'
kubectl rollout status deployment/andes-cargo-status-api -n andes-cargo --timeout=60s

What to expect:

deployment.apps/andes-cargo-status-api patched
deployment "andes-cargo-status-api" successfully rolled out

kubectl get pods -n andes-cargo -l app=andes-cargo-status-api -o wide

What to expect (something notable: by reverting the path, the template matches the original ReplicaSet's exactly again — same hash 65fcd6f6c8 — so Kubernetes doesn't create new Pods: it simply terminates the Pod with the broken probe and confirms your three original Pods, which never stopped running, stay exactly the same):

NAME                                      READY   STATUS    RESTARTS   AGE
andes-cargo-status-api-65fcd6f6c8-2bkht   1/1     Running   0          6m11s
andes-cargo-status-api-65fcd6f6c8-cnbjw   1/1     Running   0          6m5s
andes-cargo-status-api-65fcd6f6c8-m7qf8   1/1     Running   0          6m17s

Zero RESTARTS, on all three — exactly as lesson 5 predicted: a readiness failure never restarts anything, it only temporarily withdraws from traffic. That Pod was outside the Service for the entire experiment, and none of the three "real" Pods ever knew something was wrong.

Step 4 — Induce a `liveness` failure

Now the second experiment: break livenessProbe's path (leaving readinessProbe correct) to see the other consequence.

kubectl patch deployment andes-cargo-status-api -n andes-cargo --type=json \
  -p '[{"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/httpGet/path","value":"/livez-check"}]'

What to expect:

deployment.apps/andes-cargo-status-api patched

This time, since the readiness probe is correct, the new Pod does become Ready and does join the Service almost immediately — the RollingUpdate proceeds normally, replacing the three old Pods with three new Pods with the broken liveness probe. Watch the whole sequence with kubectl get pods:

kubectl get pods -n andes-cargo -l app=andes-cargo-status-api -o wide

What to expect (a few seconds after the patch — the rollout already progressed, three new Pods 1/1 Ready, RESTARTS: 0 still, because livenessProbe is just starting to be evaluated):

NAME                                      READY   STATUS    RESTARTS   AGE   IP            NODE                          NOMINATED NODE   READINESS GATES
andes-cargo-status-api-7d56987475-7pxhd   1/1     Running   0          21s   10.244.2.12   andes-cargo-cluster-worker    <none>           <none>
andes-cargo-status-api-7d56987475-kn5tb   1/1     Running   0          27s   10.244.1.9    andes-cargo-cluster-worker2   <none>           <none>
andes-cargo-status-api-7d56987475-z4f5p   1/1     Running   0          32s   10.244.2.11   andes-cargo-cluster-worker    <none>           <none>

Wait a few more seconds — initialDelaySeconds: 5 plus failureThreshold: 3 with periodSeconds: 5 mean the first restart arrives around 20 seconds into the Pod's life — and repeat the same command:

kubectl get pods -n andes-cargo -l app=andes-cargo-status-api -o wide

What to expect (RESTARTS starts climbing — each Pod, one by one on its own clock, crosses the livenessProbe threshold):

NAME                                      READY   STATUS    RESTARTS      AGE   IP            NODE                          NOMINATED NODE   READINESS GATES
andes-cargo-status-api-7d56987475-7pxhd   1/1     Running   1 (11s ago)   62s   10.244.2.12   andes-cargo-cluster-worker    <none>           <none>
andes-cargo-status-api-7d56987475-kn5tb   1/1     Running   1 (17s ago)   68s   10.244.1.9    andes-cargo-cluster-worker2   <none>           <none>
andes-cargo-status-api-7d56987475-z4f5p   1/1     Running   1 (23s ago)   73s   10.244.2.11   andes-cargo-cluster-worker    <none>           <none>

RESTARTS: 1 on all three — kubelet killed and restarted each container, with the Pod itself never disappearing (same name, same IP, same accumulated AGE). Confirm the exact cause with kubectl describe:

kubectl describe pod andes-cargo-status-api-7d56987475-z4f5p -n andes-cargo

What to expect (the Events section, literal — substitute your own Pod name):

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  79s                default-scheduler  Successfully assigned andes-cargo/andes-cargo-status-api-7d56987475-z4f5p to andes-cargo-cluster-worker
  Normal   Pulled     29s (x2 over 79s)  kubelet            spec.containers{andes-cargo-status-api}: Container image "andes-cargo-status-api:latest" already present on machine and can be accessed by the pod
  Normal   Created    29s (x2 over 79s)  kubelet            spec.containers{andes-cargo-status-api}: Container created
  Normal   Started    29s (x2 over 79s)  kubelet            spec.containers{andes-cargo-status-api}: Container started
  Warning  Unhealthy  4s (x6 over 69s)   kubelet            spec.containers{andes-cargo-status-api}: Liveness probe failed: HTTP probe failed with statuscode: 404
  Normal   Killing    4s (x2 over 59s)   kubelet            spec.containers{andes-cargo-status-api}: Container andes-cargo-status-api failed liveness probe, will be restarted

Two literal, unambiguous messages: Liveness probe failed: HTTP probe failed with statuscode: 404, followed by Container andes-cargo-status-api failed liveness probe, will be restarted. Notice the (x2 over 59s) on Pulled/Created/Started: this specific Pod already had two container starts — the original one, and a replacement after the first liveness failure — with the probe still broken, on its way to a second restart if the problem persisted. This is exactly the mechanism behind a CrashLoopBackOff: if the cause of the failure never gets fixed, Kubernetes keeps restarting, with a wait interval that grows between attempts.

Step 5 — Fix the path, and confirm the final recovery

kubectl patch deployment andes-cargo-status-api -n andes-cargo --type=json \
  -p '[{"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/httpGet/path","value":"/health"}]'
kubectl rollout status deployment/andes-cargo-status-api -n andes-cargo --timeout=90s

What to expect:

deployment.apps/andes-cargo-status-api patched
Waiting for deployment "andes-cargo-status-api" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "andes-cargo-status-api" rollout to finish: 1 old replicas are pending termination...
deployment "andes-cargo-status-api" successfully rolled out

kubectl get deployment andes-cargo-status-api -n andes-cargo
kubectl get endpoints status-api-service -n andes-cargo

What to expect (back to healthy 3/3, with three Endpoints — new Pod names and IPs versus this lesson's start, because this time there really was a complete RollingUpdate, not a simple termination like in Step 3):

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
andes-cargo-status-api   3/3     3            3           32m

Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME                 ENDPOINTS                                            AGE
status-api-service   10.244.1.10:8080,10.244.2.13:8080,10.244.2.14:8080   31m

The complete contrast, in a table

	`readiness` failure (Step 2)	`liveness` failure (Step 4)
`STATUS`	`Running`, unchanged	`Running`, unchanged
`READY`	`0/1`	`1/1` (the restart is so fast you rarely catch it at `0/1`)
`RESTARTS`	`0`, always	Climbs with each failure cycle
`Service`'s `Endpoints`	The Pod leaves, with no restart	The Pod never leaves (as long as `readiness` keeps passing)
`kubectl describe` message	`Readiness probe failed`	`Liveness probe failed` + `Killing... will be restarted`
Consequence on existing traffic	None — the `Service` already excluded it	None, as long as the Pod recovers between restarts

Common mistakes

Leaving a broken probe running too long, and ending up in a real, hard-to-exit CrashLoopBackOff (workflow, this experiment's most direct risk). What happens: someone repeats Step 4 without reverting the probe right away after confirming the first restart, and two or three failure cycles later, the restart wait interval (backoff) is already several minutes, making the fix slow to take effect. Why it happens: Kubernetes uses an exponential backoff for repeated restarts — each failure waits longer than the previous one — precisely to avoid saturating the system with constant restarts, but that means reverting late lengthens the recovery. How to spot it: if RESTARTS already shows 3 or more before you've fixed the path. How to fix it: as soon as you confirm the Liveness probe failed message in Events (as in this lesson's Step 4), revert immediately — there's no need to let the cycle repeat several times to confirm the mechanism.

Confusing Step 2's Pod "stuck at 0/1" with a failed or errored Pod (expectation). What happens: someone sees READY: 0/1 and assumes something is seriously broken, when in reality the container is perfectly healthy — its readiness probe is just configured to ask something that's never going to answer yes. How to spot it: STATUS stays at Running, RESTARTS stays at 0 — neither signal of a serious problem is present. How to fix it: READY: 0/1 with STATUS: Running and RESTARTS: 0 is, specifically, a readiness failure's signature — check the probe, not the container.

Forgetting that strategy.rollingUpdate with maxUnavailable: 0 is what protected the original Pods throughout this entire experiment (conceptual). What happens: someone repeats this same experiment on a Deployment without that explicit configuration (using Kubernetes' default, maxUnavailable: 25%) and is surprised to see that, during the readiness failure, the Service's available capacity did drop momentarily. How to spot it: compare your deployment.yaml's maxSurge/maxUnavailable against Kubernetes' default. How to fix it: maxUnavailable: 0 (used in this lesson) requires Kubernetes to keep 100% of existing replicas available during any deployment, even one that ends up failing — it's a deliberate decision in this guide so Step 2's experiment was safe to repeat with no risk to real availability.

Exercises

Exercise 1 — Reconstruct the contrast without the table. Without going back to this lesson's table, describe, in your own words, the complete difference between what you saw in Step 2 and what you saw in Step 4.

See solution

In Step 2 (readiness failure), the new Pod never made it to the Service's Endpoints — it stayed Running with READY: 0/1, RESTARTS: 0, with no consequence whatsoever for the three healthy Pods already handling traffic. In Step 4 (liveness failure), since the readiness probe did pass, the new Pods did join the Service normally — but after several consecutive liveness failures, kubelet restarted them, one by one, incrementing their RESTARTS counter with no change in the Pod's name or IP.

Exercise 2 — Explain why Step 3 created no new Pods, but Step 5 did. Both steps revert a broken probe back to /health. Explain, in two or three sentences, why Step 3 only terminated an existing Pod while Step 5 triggered a complete RollingUpdate with new Pods.

See solution

In Step 3, reverting readinessProbe back to /health made the Deployment's complete template match the original ReplicaSet's hash exactly again (the one for the three Pods that were never touched during the experiment) — Kubernetes recognized enough replicas of that exact template already existed, so it only terminated the extra Pod with the broken probe. In Step 5, on the other hand, the three "healthy" Pods at that moment already belonged to a new ReplicaSet (created during the liveness failure, with the broken probe in its template) — reverting the probe created a template hash different from those three Pods', so Kubernetes had to replace them, one by one, with a complete RollingUpdate.

Exercise 3 — Design a probe for a third scenario. andes-cargo-status-api in a future module is going to depend on a persistent database connection that takes up to 10 seconds to establish on startup. Which probe would you use to prevent Kubernetes from killing the container during those initial 10 seconds, and with which key parameter?

See solution

A startupProbe with failureThreshold × periodSeconds greater than 10 seconds — for example, periodSeconds: 2 with failureThreshold: 8 (16 seconds of margin) — pointing at the same /health or at a specific endpoint that confirms the connection is established. While startupProbe hasn't succeeded, neither readinessProbe nor livenessProbe gets evaluated — avoiding exactly the described scenario: an overly impatient livenessProbe killing a container that's merely taking time establishing its initial connection, not broken.

Summary and next step

This lesson added lesson 5's three probes to the real Deployment, and confirmed with real evidence — not just in theory — the two central consequences: a readiness failure induces READY: 0/1 with RESTARTS: 0, removing the Pod from the Service's Endpoints with no interruption for the rest; a liveness failure induces real container restarts, confirmed with RESTARTS climbing and the exact messages Liveness probe failed/Killing... will be restarted in the Events. Both experiments were done safely, protecting real availability thanks to maxUnavailable: 0, and were fully reverted before continuing.

Before moving on you should be able to: tell, with just kubectl get pods, whether a Pod is failing readiness or liveness; read a kubectl describe pod's Events to confirm the exact cause of either one; and explain why protecting maxUnavailable: 0 during a probe experiment is a safe, reversible practice.

Next lesson: HorizontalPodAutoscaler, scaling by metrics, not by gut feeling. With configuration and health resolved, the module closes with lesson 1's third question: what happens if traffic suddenly multiplies?

Resources

Kubernetes — Configure Liveness, Readiness and Startup Probes — the same reference from lesson 5, now confirmed with real evidence.
Kubernetes — Deployments: Rolling Update Deployment — official reference for maxSurge/maxUnavailable, the mechanism that made this lesson's experiments safe.
Kubernetes — Debug Running Pods — official reference for kubectl describe pod and reading Events, this lesson's central diagnostic technique.

← Previous5. Liveness, readiness, and startup probes Next →7. `HorizontalPodAutoscaler`: scaling by metrics, not by gut feeling

6. Hands-on: real probes on `status-api-service`

Description

Connection to the module

Step 1 — Add the three probes to the Deployment

Step 2 — Induce a readiness failure

Step 3 — Fix the path, and confirm the recovery

Step 4 — Induce a liveness failure

Step 5 — Fix the path, and confirm the final recovery

The complete contrast, in a table

Common mistakes

Exercises

Summary and next step

Resources

Step 1 — Add the three probes to the `Deployment`

Step 2 — Induce a `readiness` failure

Step 4 — Induce a `liveness` failure