Module 1: Why Kubernetes And The Continuity Challenge

6. Cluster architecture: control plane, nodes, and `etcd`

Next →

Description

kubectl get nodes in lesson 5 showed you three lines: one control-plane and two worker. Behind that three-line output there are, in reality, five distinct processes working together, each with a specific, limited job, none of them with full knowledge of the system on its own. This lesson opens the box: what each component does, how they communicate, and — the detail that makes the next lesson make sense — why kind runs all of this as Docker containers inside other Docker containers, with no additional magic.

Connection to the module

This lesson doesn't run any new manifest — it uses the cluster you already created in lesson 5 as live evidence. Lesson 7 picks up directly on something you're going to confirm here: the andes-cargo-status-api image, which already exists in your host's Docker, is not yet available inside the cluster — and you're going to understand exactly why before resolving it.

The control plane's five components

The control plane is the set of processes that decide what should be happening in the cluster — they never directly run your workload, they only plan and supervise it. On your andes-cargo-cluster cluster, all five run together on the andes-cargo-cluster-control-plane node:

  • kube-apiserver — the single entry point to the cluster. Everything — kubectl, any operator like ArgoCD (Module 5), any admission controller (Module 6) — talks to Kubernetes exclusively through this component, never directly with etcd or with any other process. It validates every request, authenticates it, authorizes it, and — most important for Module 6 — it's the exact point where admission control runs before an object gets saved.
  • etcd — the cluster's database, and the only source of truth about the current state of everything. When you create a Deployment, that YAML document ends up, after passing through kube-apiserver, saved as a record in etcd. Nothing in Kubernetes "remembers" anything on its own outside of etcd — if etcd is lost without a backup, the cluster loses its entire memory of what existed.
  • kube-scheduler — decides which node a new Pod should run on, based on each node's available resources and on the constraints the Pod itself declares (for example, how much CPU/memory it needs — something you explicitly configure in Module 3). The scheduler doesn't run anything — it only assigns, and tells kube-apiserver what its decision was.
  • kube-controller-manager — the process that runs, under the hood, dozens of different control loops, each one continuously comparing "what exists right now" against "what should exist" (what's declared in etcd), and acting to close any gap without anyone asking. This is the exact mechanism that makes a ReplicaSet replace a deleted Pod with no human intervention — you're going to see it in action firsthand in Module 2.
  • kube-proxy — runs on every node (not only the control plane), and maintains the networking rules that make it possible for a Kubernetes Service (Module 2) to distribute traffic across several Pods, no matter which node each one is running on.

And, on every node — control plane and workers alike — a sixth process runs, different in nature from the five above because it doesn't live "in the cluster" as an object but directly in the node's operating system:

  • kubelet — the agent that actually executes something. It receives instructions from kube-apiserver ("this Pod should run here"), and uses the node's container engine (in kind, containerd) to start it, monitor its health, and report the real state back. If kube-scheduler is who decides, kubelet is who executes the decision.

The complete diagram

                     andes-cargo-cluster — REAL ARCHITECTURE

  ┌──────────────────────────────────────────────────────────────────┐
  │  NODE: andes-cargo-cluster-control-plane                             │
  │                                                                        │
  │   kubectl  ───────▶  ┌─────────────────┐                              │
  │   (your terminal)      │  kube-apiserver   │ ◀── the single entry point │
  │                       └────────┬────────┘                              │
  │                                 │                                       │
  │            ┌────────────────────┼────────────────────┐                 │
  │            ▼                    ▼                    ▼                 │
  │   ┌────────────────┐  ┌──────────────────┐  ┌──────────────────────┐ │
  │   │  etcd             │  │  kube-scheduler     │  │  kube-controller-       │ │
  │   │  (saved state)     │  │  (decides WHICH NODE)│  │  manager (reconciles)  │ │
  │   └────────────────┘  └──────────────────┘  └──────────────────────┘ │
  │                                                                        │
  │   kubelet + kube-proxy (also run here, as on every node)                │
  └──────────────────────────────────────────────────────────────────┘
                                    │
                     kube-apiserver tells each kubelet
                     which Pods should run on its node
                                    │
        ┌───────────────────────────┼───────────────────────────┐
        ▼                                                       ▼
  ┌──────────────────────────┐                       ┌──────────────────────────┐
  │  NODE: ...-worker            │                       │  NODE: ...-worker2          │
  │                             │                       │                             │
  │   kubelet  ──▶ containerd     │                       │   kubelet  ──▶ containerd     │
  │   kube-proxy                  │                       │   kube-proxy                  │
  │   (your Pods will run             │                       │   (your Pods can also            │
  │    here starting in Module 2)      │                       │    run here)                     │
  └──────────────────────────┘                       └──────────────────────────┘

kube-apiserver is, literally, the only component any other piece — including kubectl, including every kubelet — is allowed to talk to directly. Neither kube-scheduler nor kube-controller-manager touch etcd directly: both query and write through kube-apiserver. This apparently rigid centralization is what makes possible the complete security model Module 6 builds — if there's only one door, only one door needs watching.

Real evidence: the five components, running as Pods

Here's this lesson's most important detail, and the reason kind is an honest learning tool and not a simulator: in Kubernetes, even the control plane itself runs as Pods, with the same mechanism you're going to use for your own workload starting in Module 2. Confirm it yourself, against the cluster you already have up:

kubectl get pods -A -o wide

What to expect (Pod names with a hash suffix — coredns-589f44dc88-2mqbh, for example — are variable by design; the prefixes, the NAMESPACE, and what runs on which node are literal for this architecture):

NAMESPACE            NAME                                                        READY   STATUS    RESTARTS   AGE    NODE
kube-system          coredns-589f44dc88-2mqbh                                    1/1     Running   0          100s   andes-cargo-cluster-control-plane
kube-system          coredns-589f44dc88-q6qs4                                    1/1     Running   0          100s   andes-cargo-cluster-control-plane
kube-system          etcd-andes-cargo-cluster-control-plane                      1/1     Running   0          108s   andes-cargo-cluster-control-plane
kube-system          kindnet-4z9xs                                               1/1     Running   0          95s    andes-cargo-cluster-worker2
kube-system          kindnet-8njwk                                               1/1     Running   0          101s   andes-cargo-cluster-control-plane
kube-system          kindnet-zffz9                                               1/1     Running   0          95s    andes-cargo-cluster-worker
kube-system          kube-apiserver-andes-cargo-cluster-control-plane            1/1     Running   0          108s   andes-cargo-cluster-control-plane
kube-system          kube-controller-manager-andes-cargo-cluster-control-plane   1/1     Running   0          109s   andes-cargo-cluster-control-plane
kube-system          kube-proxy-2xk6h                                            1/1     Running   0          95s    andes-cargo-cluster-worker
kube-system          kube-proxy-nqqvw                                            1/1     Running   0          95s    andes-cargo-cluster-worker2
kube-system          kube-proxy-w2scb                                            1/1     Running   0          101s   andes-cargo-cluster-control-plane
kube-system          kube-scheduler-andes-cargo-cluster-control-plane            1/1     Running   0          108s   andes-cargo-cluster-control-plane
local-path-storage   local-path-provisioner-855c7b7774-h2km4                     1/1     Running   0          100s   andes-cargo-cluster-control-plane

Notice the naming pattern: etcd-andes-cargo-cluster-control-plane, kube-apiserver-andes-cargo-cluster-control-plane, kube-scheduler-..., kube-controller-manager-... — the control plane's four central components, running as Pods, all on the control-plane node, exactly where this lesson's diagram placed them. kube-proxy and kindnet (the CNI kind uses by default) each appear three times — one per node in the cluster, confirming that those two do run everywhere, not just on the control plane. And coredns — the internal name server you confirmed in lesson 5 — runs as two separate Pods, for redundancy.

No component on this list was "invented" by kind — they're, literally, the same container images (registry.k8s.io/kube-apiserver, registry.k8s.io/etcd, etc.) that run inside a real EKS node. Confirm it with crictl, the node's internal container engine's command-line tool (not Docker's):

docker exec andes-cargo-cluster-control-plane crictl images

What to expect (the sizes are literal for this exact version; they may vary slightly across processor architectures):

IMAGE                                           TAG                  IMAGE ID            SIZE
docker.io/kindest/kindnetd                      v20260528-9350166c   f2ede2b789a61       35.6MB
docker.io/kindest/local-path-helper             v20260131-7181c60a   8f375e9f93513       2.74MB
docker.io/kindest/local-path-provisioner        v20260521-9fb22683   3501a03785a84       14.2MB
registry.k8s.io/coredns/coredns                 v1.14.2              fe81a497e85f1       20.8MB
registry.k8s.io/etcd                            3.6.8-0              6da6ea097b384       21.1MB
registry.k8s.io/kube-apiserver                  v1.36.1              4923943f21256       89.9MB
registry.k8s.io/kube-controller-manager         v1.36.1              39d983367f38c       80.2MB
registry.k8s.io/kube-proxy                      v1.36.1              01ad784c02283       80.9MB
registry.k8s.io/kube-scheduler                  v1.36.1              76e62361b06b5       57.3MB
registry.k8s.io/pause                           3.10                 afb61768ce381       268kB

etcd, marked with its own version (3.6.8-0) independent from Kubernetes' overall version (v1.36.1) — a detail that confirms etcd is a separate project, with its own release cycle, that Kubernetes adopts as a dependency.

Why kind runs all of this on nested Docker containers

This is the lesson's most important detail, and the exact reason for a behavior you're going to resolve in lesson 7. Each "node" in your cluster — andes-cargo-cluster-control-plane, andes-cargo-cluster-worker, andes-cargo-cluster-worker2 — is, from your host Docker's side, a single container (you confirmed this with docker ps in lesson 5). But inside that container runs its own independent container engine — containerd — which is what kubelet actually uses to start every Pod. It's, literally, a container engine running inside another container: your host's Docker sees a single process called andes-cargo-cluster-control-plane; that process, internally, administers its own collection of "containers" (the control plane Pods you just listed) with a containerd your host's Docker can't see directly.

The practical consequence, which you confirm yourself right now:

docker images andes-cargo-status-api --format "table {{.Repository}}\t{{.Tag}}\t{{.ID}}"

What to expect (if you already built the image in aws-serverless-and-containers-guide or in lesson 7 of this module; your IMAGE ID will be different — literal in structure, variable in the exact ID):

REPOSITORY               TAG       IMAGE ID
andes-cargo-status-api   latest    d07c06907665

And now, the same question, but inside the node's containerd:

docker exec andes-cargo-cluster-control-plane crictl images | grep andes-cargo-status-api

What to expect (no output — the image doesn't show up, even though you just confirmed it does exist in your host's Docker):

(no output)

There's the exact problem lesson 7 resolves: your host's Docker and each node's internal containerd are two completely separate image caches, even though both ultimately run on the same physical machine. Building an image with docker build makes it available for your host's Docker — never, automatically, for a kind node's internal containerd. That's exactly why the kind load docker-image command exists, the star of the next lesson.

Analogy: the stadium's control room

Picking back up lesson 5's scale-model stadium: if the whole stadium is your andes-cargo-cluster cluster, the control plane is the stadium's control room — the place, physically separate from the stands and the playing field, where a small team decides what should be happening (kube-scheduler deciding which section to seat each new group of spectators in), keeps the official record of every assigned seat (etcd, the master ledger), and constantly watches that reality matches what was planned, correcting it without anyone asking if something drifts (kube-controller-manager). kube-apiserver is that control room's single window that serves the public — no one walks in to talk directly to the master ledger or to the assignment team, everything goes through that single window. And kubelet, in each section of the stands, is the on-site staff that actually carries out the instructions coming down from the control room — opens the right door, seats the people, reports if a seat is left empty.

Common mistakes

Assuming etcd is "just another database" that can be swapped for any other with no consequence (conceptual). What happens: someone, familiar with general-purpose relational or document databases, underestimates etcd's role and thinks of it as an interchangeable implementation detail. Why it happens: "database" sounds generic, and it's easy to miss that Kubernetes depends on very specific consistency guarantees that etcd (a distributed key-value store, with consensus via the Raft algorithm) provides by design. How to spot it: if you believe losing etcd without a backup would be "recoverable by checking the Pods that are still running." How to fix it: etcd is the only source of truth for the cluster's desired state — if it's lost without a backup, the cluster loses its entire memory of what should exist, even though Pods that were already running might keep living for a while. On real EKS (Module 7), AWS administers and backs up etcd for you — one of the concrete reasons the managed control plane has its own cost.

Confusing kube-scheduler with the process that "runs" Pods (conceptual, this lesson's most common mistake). What happens: someone reads "the scheduler decides where a Pod runs" and assumes the scheduler also starts it. Why it happens: in everyday language, "deciding" and "executing" blend together easily. How to spot it: if you can't explain, without hesitating, the difference in work between kube-scheduler and kubelet. How to fix it: kube-scheduler only assigns — it decides which node a new Pod should run on, and reports that to kube-apiserver — the one that actually starts it, monitors it, and reports its real status back is kubelet, running on the specific node the Pod was assigned to. They're two separate responsibilities, in two different components, on purpose.

Trying to debug an image problem with docker images when the problem is inside the cluster (workflow, directly related to this lesson's nested-containers section). What happens: someone, in a later module, has a Pod with an ImagePullBackOff error, runs docker images on their host, sees the image does exist there, and — mistakenly — concludes the problem can't be about the image. Why it happens: without having internalized this lesson's distinction between the host's Docker and each node's internal containerd, it's natural to assume "the image exists on my machine" is enough. How to spot it: if your evidence that "the image is available" comes only from docker images, without also having confirmed with crictl images inside the specific node. How to fix it: always use docker exec <node-name> crictl images to confirm which images are really available inside the cluster — lesson 7 of this module installs the full habit.

Exercises

Exercise 1 — Match each component to its job. Without looking at the lesson, pair each of these five components with its responsibility: kube-apiserver, etcd, kube-scheduler, kube-controller-manager, kubelet. Responsibilities: (a) runs the Pod on the assigned node; (b) stores the cluster's desired state; (c) decides which node a new Pod should run on; (d) is the cluster's single entry point; (e) continuously reconciles reality against what's declared.

See solution

kube-apiserver → (d); etcd → (b); kube-scheduler → (c); kube-controller-manager → (e); kubelet → (a).

Exercise 2 — Explain why kube-apiserver is a centralization point, not a conceptual bottleneck. In two or three sentences, explain why the "single entry point" (kube-apiserver) design is a security decision, not just a technical limitation.

See solution

A complete answer sounds roughly like this: "If any component could write directly to etcd or talk directly to other internal components, there would be no single point where authentication, authorization, and validation could be applied consistently. By forcing absolutely everything — kubectl, an operator like ArgoCD, any admission controller — to go through kube-apiserver, Kubernetes guarantees every request passes through the same set of controls, with no exceptions. This is exactly what makes Module 6's admission control possible: if there were multiple doors, an admission webhook could only watch one of them."

Exercise 3 — Predict the result of a new command. Without running it yet, predict what you'd expect to see if you ran docker exec andes-cargo-cluster-worker crictl images (note: worker, not control-plane) against your current cluster. Would you expect to see the same images as on the control plane?

See solution

Not exactly the same — you'd expect to see kindnetd, local-path-helper (if that node ever needed it) and pause, because kube-proxy and kindnet run on every node, but you would not expect to see etcd, kube-apiserver, kube-scheduler, or kube-controller-manager, because those four components only run as Pods on the control-plane node, never on worker nodes. This is exactly the pattern you confirmed with kubectl get pods -A -o wide in this lesson: the NODE column showed those four Pods only on andes-cargo-cluster-control-plane.

Summary and next step

In this lesson you opened up the cluster you created in lesson 5: five control plane components (kube-apiserver as the single entry point, etcd as the only source of truth, kube-scheduler deciding where, kube-controller-manager reconciling with no one asking) plus kubelet and kube-proxy running on every node, confirmed with real evidence from your own cluster (kubectl get pods -A, crictl images). And you discovered, with literal evidence, the detail that makes the next lesson possible: your host's Docker and each kind node's internal containerd are completely separate image caches — building an image doesn't automatically make it available inside the cluster.

Before moving on you should be able to: name the control plane's five components and each one's exact responsibility; explain why kube-apiserver is the single entry point; and predict, without running it, which kube-system Pods you'd expect to see on a worker node versus a control-plane one.

Lesson 7 resolves, explicitly and with executed evidence, the gap you just confirmed: how to get the andes-cargo-status-api image from your host's Docker into the cluster, with kind load docker-image.

Resources

  1. Kubernetes — Kubernetes Components — complete official reference for the five control-plane and node components covered in this lesson.
  2. Kubernetes — Kubernetes Architecture — the official overview of how these components relate to each other.
  3. etcd — What is etcd? — official documentation for the etcd project, independent from Kubernetes but adopted as its state store.
  4. kind — Design Principles — the project's official explanation of why each node runs its own internal container engine.
  5. Kubernetes — Debugging Kubernetes Nodes With crictl — official reference for crictl, used in this lesson to inspect a node's internal containerd.
← Previous5. Hands-on: your first clusterNext →7. Hands-on: loading the `andes-cargo-status-api` image into the cluster