5. Liveness, readiness, and startup probes

Description

Today, andes-cargo-status-api has three replicas with STATUS: Running — but "the process started" and "the process is in condition to handle a real request" are different claims, and Kubernetes, up to this point in the guide, has no way to tell them apart. If one of the three Pods hung right now — without crashing, just no longer responding — kubectl get pods would keep showing Running, and the Service would keep sending it traffic indefinitely. This lesson resolves exactly that blind spot with three health-verification mechanisms, each one answering a different question.

Connection to the module

This is the second of the three pieces lesson 1 promised. Lesson 6 adds real probes to the Deployment and induces, on purpose, a failure of each type — watching with real evidence what Kubernetes does in each case.

---

The exact problem: Running doesn't mean Healthy

STATUS: Running, the column you already know from every kubectl get pods in this guide, answers a single question: does the container have a live main process? That question gets answered at the operating-system level — kubelet checks that the process with PID 1 inside the container hasn't exited — with no notion whatsoever of what that process does or whether it does it correctly. A Flask server stuck in an infinite loop, responding to no request, or one that lost its connection to a critical database, still has a live process — it's still Running, indefinitely, as long as no one asks it anything more specific.

Kubernetes offers three mechanisms to ask more specific questions, each with a different consequence when the answer is "no":

Probe	Question it asks	What Kubernetes does if it fails
startupProbe	Has this application finished starting up yet?	Nothing else runs (neither liveness nor readiness) until this probe succeeds, or until it exhausts its own failureThreshold (in which case, it restarts the container)
readinessProbe	Is this replica ready to receive traffic right now?	Removes the Pod from the Service's Endpoints list — the container keeps running, no restart, it just stops receiving new traffic
livenessProbe	Is this process still alive, or is it so broken it needs replacing?	Restarts the container (the same mechanism that triggers a CrashLoopBackOff if the problem persists)

---

Analogy: the medical checkup, three different questions

Think of each Pod as an employee showing up for work. A startupProbe is the question asked on day one: "have you finished settling into your desk, is your computer on, and are your credentials working?" — until the answer is yes, no one asks anything else, nor is the employee expected to serve any customer yet. A readinessProbe is the question asked every few minutes throughout the day: "can you serve a customer right now?" — if the answer is no (they're on a long call, in the bathroom, in a meeting), new customers simply get routed to another employee, with no one telling them they're fired; the moment they answer yes again, they resume serving customers with total normalcy. A livenessProbe is a much more serious question: "are you still yourself, or does someone need to call in a replacement because something went deeply wrong?" — if the answer is no, consistently, the organization doesn't wait: it replaces that person with a new one, hoping the new one does work.

---

A probe's anatomy

All three probes share the same field structure — the name changes (startupProbe/readinessProbe/livenessProbe), but the parameters are identical:

readinessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 2
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 1
  successThreshold: 1

• The verification mechanism (httpGet here — Kubernetes also offers tcpSocket, to confirm only that a port accepts connections, and exec, to run a command inside the container and evaluate its exit code). This guide uses httpGet against /health, the same endpoint app.py already exposes since aws-serverless-and-containers-guide, Module 6 — an endpoint that responds 200 with no dependency on any external data, exactly the kind of lightweight check a probe needs.
• initialDelaySeconds — how long kubelet waits after the container starts before doing the first check.
• periodSeconds — how often, in seconds, it repeats the check.
• timeoutSeconds — how long it waits for a response before counting that attempt as failed.
• failureThreshold — how many consecutive failures it takes to consider the probe "failed" (and trigger the consequence from the previous table).
• successThreshold — how many consecutive successes it takes to consider it "successful" again after failing (default 1 for readiness/startup; liveness always uses 1, by Kubernetes design).

---

Why confusing readiness with liveness is this layer's most common mistake

The most frequent confusion in real clusters — and the reason this lesson devotes a full section to it before the hands-on part — is using the same probe with the same consequence for both cases, without telling "temporarily busy" apart from "genuinely broken." A real, common example: an application that, under high load, takes a few extra seconds to respond. If that slowness is measured with an overly strict livenessProbe (low timeoutSeconds, low failureThreshold), Kubernetes is going to restart a process that wasn't broken — it was just busy — worsening the problem: fewer available replicas right when they're most needed, each restart adding its own startup time to the already-existing pressure. That same scenario, measured with a readinessProbe, simply, temporarily removes the busy Pod from the traffic list — with no restart — letting it recover on its own and rejoin as soon as it can respond again.

The practical rule this guide follows: readinessProbe should be more sensitive (fails faster) than livenessProbe (fails slower, with more margin). Temporarily removing a Pod from the Service is a cheap, reversible operation; restarting a container is an expensive one and, if the cause doesn't resolve itself, it can enter a restart cycle (CrashLoopBackOff) that worsens availability instead of improving it. Lesson 6 uses exactly this asymmetry: readinessProbe with failureThreshold: 1 (fails on the first failed attempt), livenessProbe with failureThreshold: 3 (needs three consecutive failures before restarting).

              THE THREE PROBES, IN A CONTAINER'S LIFECYCLE

  Container starts
        │
        ▼
  ┌─────────────────┐   repeated failure   ┌──────────────────┐
  │  startupProbe    │ ───────────────────▶│  restarts the      │
  │  "did it start?" │                      │  container          │
  └────────┬─────────┘                      └──────────────────┘
           │ success
           ▼
  ┌──────────────────────┐        ┌──────────────────────┐
  │  readinessProbe         │        │  livenessProbe           │
  │  "ready RIGHT NOW?"     │        │  "really still alive?"    │
  │  runs in parallel        │        │  runs in parallel          │
  │  with liveness, always   │        │  with readiness, always    │
  └──────────┬─────────────┘        └───────────┬──────────────┘
             │ fails                              │ repeated failure
             ▼                                    ▼
   leaves the Service's              kubelet restarts
   Endpoints — NO restart            the container
   (Common mistakes, lesson 6)       (Common mistakes, lesson 6)

---

Common mistakes

Using the same endpoint and the same thresholds for readiness and liveness, without thinking about the asymmetry (conceptual, this lesson's most important one). What happens: someone copies the exact same readinessProbe configuration to livenessProbe, reasoning "if the endpoint is fine for one, it's fine for the other." Why it happens: it technically works, and produces no immediate error — the problem only shows up under real load conditions or transient slowness, when both probes fail at the same time and Kubernetes restarts Pods that were only busy, not broken. How to spot it: if your livenessProbe has a failureThreshold equal to or lower than your readinessProbe's. How to fix it: this lesson's rule — readiness sensitive and cheap to fail (leaves traffic, recovers on its own), liveness with more margin and expensive to fail (restarts a process). Lesson 6 applies this asymmetry with real evidence.

Pointing a probe at an endpoint that depends on external services (design, a subtle mistake). What happens: someone configures livenessProbe against an endpoint that, like /shipments/<id> at this point in the guide, depends on an external database — and the moment that external database has a transient problem, Kubernetes starts restarting healthy containers that merely depend on a downed third party, solving nothing (restarting the process doesn't fix the database). How to spot it: if your probe endpoint makes any network call outside the process itself. How to fix it: a probe endpoint should verify only the process's own health — exactly /health's design in app.py, which responds 200 with no external dependency whatsoever — never the health of a system it indirectly depends on.

Forgetting the three probes run continuously, not just once (expectation). What happens: someone assumes that, once a Pod passes its readinessProbe the first time, it's "approved" forever and the probe stops running. Why it happens: the name "startup probe" (startupProbe) does work that way — once, until the first success — and it's easy to generalize that idea to the other two. How to spot it: if you're surprised to see a Pod that had been Ready for hours suddenly leave the Endpoints list. How to fix it: readinessProbe and livenessProbe run every periodSeconds, for the Pod's entire life, with no exception — a Pod that was healthy for hours can fail its readinessProbe at any moment if it stops responding correctly, exactly the scenario lesson 6 is going to induce on purpose.

---

Exercises

Exercise 1 — Fill in the three probes' table from memory. Without going back to the corresponding section, for each of the three probes, write the question it asks and the exact consequence of it failing.

See solution

Probe	Question	Consequence of failing
startupProbe	Has it finished starting up?	Blocks readiness/liveness until success, or restarts if it exhausts its own threshold
readinessProbe	Is it ready for traffic now?	Leaves the Service's Endpoints, no restart
livenessProbe	Is it really still alive?	kubelet restarts the container

Exercise 2 — Diagnose a poorly designed probe setup. A team configures livenessProbe with failureThreshold: 1 and timeoutSeconds: 1 against an endpoint that makes a slow database query. Explain, in two or three sentences, what's going to happen under high load, and why it's a design mistake.

See solution

Under high load, the database query probably takes more than 1 second to respond at least once — with failureThreshold: 1, a single failure (which could just be transient slowness, not a broken process) is enough for Kubernetes to restart the container. The design mistake is twofold: using an endpoint that depends on a slow external system for liveness (instead of a lightweight /health), and giving an almost-zero failure margin (failureThreshold: 1) to a probe whose consequence is the most expensive of the three (restarting a process). The real result is restarting healthy processes at the worst possible moment — under high load — reducing available capacity exactly when it's most needed.

Exercise 3 — Explain the medical checkup analogy without using "readiness" or "liveness." In two or three sentences, without naming any Kubernetes technical term, explain the difference between this lesson's two continuous health questions, using the employee analogy.

See solution

A reasonable answer: "One question gets asked all day, every few minutes: 'can you serve a customer right now?' — if the answer is no for a moment (they're in the bathroom, on a call), customers simply get routed to another coworker, with no serious consequence, and the moment they're available again they resume their work normally. The other question is much more serious and gets answered with much more margin before acting: 'are they still the same person, or do they need to be replaced entirely?' — that decision isn't made at the first sign of doubt, only after several confirmations that something is genuinely wrong."

---

Summary and next step

This lesson resolved the blind spot the previous modules' Deployment left: STATUS: Running only confirms a process is alive, not that it's in condition to handle traffic. The three probes — startupProbe, readinessProbe, livenessProbe — answer different questions, with different consequences: the first blocks the rest until it starts up; the second removes the Pod from the Service without restarting it; the third restarts the container. This lesson's hard rule — readiness sensitive and cheap to fail, liveness with more margin and expensive to fail — exists precisely to avoid this layer's most common mistake: confusing "temporarily busy" with "genuinely broken."

Before moving on you should be able to: explain the exact difference between the three probes, without confusing their consequences; justify why readiness should have a lower failureThreshold than liveness; and explain why a probe should never depend on an external system.

Next lesson: hands-on, real probes on status-api-service. There you add the three probes to the real Deployment, and induce — on purpose, in a controlled way — a readiness failure and a liveness failure, seeing the exact difference with your own eyes.

Resources

1. Kubernetes — Configure Liveness, Readiness and Startup Probes — the complete official guide, with examples of all three verification mechanisms (httpGet, tcpSocket, exec).
2. Kubernetes — Pod Lifecycle: Container probes — official reference for the complete lifecycle and the interaction between all three probes.
3. aws-serverless-and-containers-guide (NIEVA), Module 6, lesson 5 — the /health endpoint, unchanged, which this guide reuses as all three probes' check.