Module 8: Capstone Andes Cargo On Kubernetes

4. End-to-end walkthrough: a change the gate stops

Description

Module 6 tested Gatekeeper and Kyverno against isolated test Pods, applied with direct kubectl apply. This lesson raises the stakes: a real attempt to strip resource limits off the actual production Deployment, andes-cargo-status-api, pushed through the same GitOps path lesson 3 used for the change that did pass. The result, captured live against this same cluster, confirms half of this guide's promise — the gate stops the change, before any new Pod exists — and reveals something no previous module showed: the two engines don't cover exactly the same territory, and this lesson is the first place where that difference becomes visible with real evidence.

Connection to the module

This lesson puts lesson 2's sequence diagram's else branch to the test — the object violates a policy. Lesson 3 tested the alt branch. Both share exactly the same mechanism (Git → ArgoCD → admission control); the only thing that changes is the change's content.

Step 0 — The scenario: a real reason, a dangerous change

An Andes Cargo engineer, investigating an intermittent OOMKilled on andes-cargo-status-api, decides to temporarily "loosen" the memory limits while diagnosing — a real shortcut, the kind any team under incident pressure has considered at some point. Instead of adjusting the value, in a rush, they remove the entire resources block:

grep -n "resources:" -A6 deployment.yaml

What to expect (the state before this lesson's change — the same block lesson 3 confirmed intact):

34:          resources:
35:            requests:
36:              cpu: "100m"
37:              memory: "64Mi"
38:            limits:
39:              cpu: "250m"
40:              memory: "128Mi"

Step 1 — The change: remove the entire `resources` block

# deployment.yaml (fragment, BEFORE → AFTER this change)
        - name: andes-cargo-status-api
          image: andes-cargo-status-api:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          envFrom:
            - configMapRef:
                name: andes-cargo-status-api-config
            - secretRef:
                name: andes-cargo-status-api-secrets
-         resources:
-           requests:
-             cpu: "100m"
-             memory: "64Mi"
-           limits:
-             cpu: "250m"
-             memory: "128Mi"
          startupProbe:
            httpGet:
              path: /health

git diff

What to expect (literal, executed):

diff --git a/deployment.yaml b/deployment.yaml
index 0dce5da..4d874d8 100644
--- a/deployment.yaml
+++ b/deployment.yaml
@@ -33,13 +33,6 @@ spec:
                 name: andes-cargo-status-api-config
             - secretRef:
                 name: andes-cargo-status-api-secrets
-          resources:
-            requests:
-              cpu: "100m"
-              memory: "64Mi"
-            limits:
-              cpu: "250m"
-              memory: "128Mi"
           startupProbe:
             httpGet:
               path: /health

Seven lines removed, exactly the block Module 6, lesson 4, requires on every Pod in the andes-cargo namespace — this is, deliberately, the same violation bad-pod-no-limits.yaml tested there, this time against the real Deployment, not a test Pod.

This git push still runs from your machine against localhost:3000, same as lesson 3 — if Gitea's port-forward closed between lessons, open it again before continuing: kubectl port-forward -n gitea svc/gitea-http 3000:3000.

git add deployment.yaml
git commit -m "Temporarily remove resource limits from andes-cargo-status-api to debug OOM"
git push origin main

What to expect (literal, executed — the hash is your variable value):

[main 229cd7b] Temporarily remove resource limits from andes-cargo-status-api to debug OOM
 1 file changed, 7 deletions(-)
To http://localhost:3000/andes-cargo/andes-cargo-k8s.git
   d0b98c6..229cd7b  main -> main

Step 2 — ArgoCD detects it, and gets stuck: `OutOfSync`, not `Synced`

for i in $(seq 1 24); do
  ts=$(date +%H:%M:%S)
  rev=$(kubectl get application andes-cargo-status-api -n argocd -o jsonpath='{.status.sync.revision}' | cut -c1-7)
  sync=$(kubectl get application andes-cargo-status-api -n argocd -o jsonpath='{.status.sync.status}')
  echo "$ts rev=$rev sync=$sync"
  sleep 10
done

What to expect (literal, executed — unlike lesson 3, where sync went straight to Synced, here it stays OutOfSync indefinitely):

17:23:10 rev=d0b98c6 sync=Synced
17:23:20 rev=d0b98c6 sync=Synced
...
17:24:31 rev=d0b98c6 sync=Synced
17:24:41 rev=229cd7b sync=OutOfSync
17:24:48 rev=229cd7b sync=OutOfSync
17:24:56 rev=229cd7b sync=OutOfSync
...
17:26:43 rev=229cd7b sync=OutOfSync

This is the first signal something different happened compared to lesson 3: there, revision advanced and sync went back to Synced in the same cycle. Here, revision advances (ArgoCD did detect commit 229cd7b), but sync stays OutOfSync — selfHeal: true is trying to apply the change, and something is repeatedly stopping it.

Step 3 — The literal message: Kyverno blocked the `Deployment`, before it got saved

kubectl logs -n argocd argocd-application-controller-0 --since=5m | grep -A3 "denied the request"

What to expect (literal, executed — this is this lesson's central finding, captured directly from ArgoCD's real attempt):

"message":"Updating operation state. phase: Running -> Failed, message: 'waiting for healthy state of
/Namespace/andes-cargo and 8 more resources' -> 'one or more synchronization tasks completed
unsuccessfully, reason: error when patching \"/dev/shm/793282161\": admission webhook
\"validate.kyverno.svc-fail\" denied the request: 

resource Deployment/andes-cargo/andes-cargo-status-api was blocked due to the following policies 

andes-cargo-require-resource-limits:
  autogen-require-resource-requests-and-limits: 'validation error: every container
    must set resources.requests and resources.limits for cpu and memory. rule
    autogen-require-resource-requests-and-limits failed at path
    /spec/template/spec/containers/0/resources/limits/''"

Read this message with the same discipline Module 6, lesson 4, taught: admission webhook "validate.kyverno.svc-fail" denied the request confirms Kyverno, not Gatekeeper, is who stopped this attempt — and Deployment/andes-cargo/andes-cargo-status-api confirms something new compared to everything you saw in Module 6: the policy rejected the Deployment object directly, not a Pod. The name of the rule that fired, autogen-require-resource-requests-and-limits, carries the autogen- prefix for a concrete reason this lesson's Step 5 explains in depth.

ArgoCD tried three times (syncId: 00023, 00024, 00025 in the controller's complete log), and got the same rejection all three times — it's not a transient error, it's the same reason, repeated, because the object Git declares keeps violating the policy while the file doesn't change.

Step 4 — Confirm: the real `Deployment` never changed

kubectl get deployment andes-cargo-status-api -n andes-cargo -o jsonpath='{.spec.template.spec.containers[0].resources}{"\n"}'
kubectl get pods -n andes-cargo -l app=andes-cargo-status-api

What to expect (literal, executed — the resources block is still complete, and the Pods are still the same ones from lesson 3, with no new ReplicaSet):

{"limits":{"cpu":"250m","memory":"128Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}

NAME                                      READY   STATUS    RESTARTS   AGE
andes-cargo-status-api-669755d655-np5mt   1/1     Running   0          6m46s
andes-cargo-status-api-669755d655-qhgwx   1/1     Running   0          6m40s

This is the exact point this lesson's task asked you to verify: the rejected object never reached etcd. The live Deployment still declares the previous commit's (d0b98c6) complete resources, because the attempt to apply 229cd7b got rejected by Kyverno's admission webhook before kube-apiserver could save the change — Git says one thing (229cd7b, no resources), the real cluster says another (d0b98c6, with resources), and that difference is exactly what Sync Status: OutOfSync is reporting.

Step 5 — The real finding: Gatekeeper never got to evaluate this attempt

Before repeating the test with Gatekeeper, it's worth confirming something Step 3's message never said at any point: did Gatekeeper participate in this rejection, at all?

kubectl logs -n gatekeeper-system deploy/gatekeeper-controller-manager --since=6m | grep "andes-cargo-status-api"

What to expect (literal — no output line at all):

Zero lines. Unlike Module 6, lesson 8 (where both engines evaluated the same Pod, and only Gatekeeper's message showed up in kubectl while the logs confirmed Kyverno also acted), here Gatekeeper was never invoked at all for this attempt — it's not that it lost the race against Kyverno, it's that its Constraint doesn't apply to this kind of object. Confirm it by reviewing the Constraint's exact definition (Module 6, lesson 4):

kubectl get k8srequiredresources andes-cargo-must-have-resource-limits -o jsonpath='{.spec.match.kinds}{"\n"}'

What to expect (literal, executed):

[{"apiGroups":[""],"kinds":["Pod"]}]

There's the exact cause: this Constraint's spec.match.kinds declares, unambiguously, ["Pod"] — nothing else. A Deployment isn't a Pod; it's a different object, with its own apiGroup (apps), that this Constraint never declared within its scope. When ArgoCD tried to apply the modified Deployment, kube-apiserver invoked Gatekeeper's ValidatingWebhookConfiguration same as always — but Gatekeeper, when checking its own rules, found none applies to an object of type Deployment, and responded allowed: true without evaluating anything further.

Kyverno, on the other hand, did catch it — and the reason is in the rule name you saw in Step 3: autogen-require-resource-requests-and-limits. Module 6, lesson 6's ClusterPolicy declared spec.background: true and a rule written for kind: Pod — but Kyverno, seeing that background: true, automatically generates equivalent rules for the most common controllers that produce Pods (Deployment, ReplicaSet, DaemonSet, StatefulSet, Job, CronJob), applying the same pattern to the corresponding field inside spec.template.spec.containers[].resources of each — with no one in Module 6 having written that additional rule by hand. That's, literally, Kyverno's autogen mechanism, and it's the exact reason Kyverno stopped this attempt at the earliest possible point — the Deployment itself — while Gatekeeper, with the Constraint configured the way Module 6 left it, would have waited for a real Pod to exist before acting.

             WHY ONLY ONE ENGINE STOPPED THIS ATTEMPT

  ArgoCD tries to apply: Deployment/andes-cargo-status-api (no resources)
                          │
              ┌────────────┴────────────┐
              ▼                           ▼
   Gatekeeper Constraint          Kyverno ClusterPolicy
   match.kinds: ["Pod"]           background: true (generates
   Deployment ≠ Pod                autogen- rules for Deployment/RS/etc.)
              │                           │
   "this isn't mine               "this IS mine, the autogen-
    to evaluate"                   rule DOES cover Deployment"
              │                           │
      allowed: true                allowed: false — DENIED
              │                           │
              └────────────┬────────────┘
                           ▼
              kube-apiserver: DENIED (one denial is enough)
              The Deployment never gets saved in etcd

Step 6 — Reproduce the "side by side" comparison Module 6 showed: a bare Pod, with no `Deployment` in the way

To confirm Gatekeeper does protect against this same violation — just at a different scope (Pod, not Deployment) — repeat Module 6's experiment against a test Pod, exactly the same way as there:

cat > bad-pod-no-limits-m8.yaml << 'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: bad-pod-no-limits-m8
  namespace: andes-cargo
spec:
  containers:
    - name: bad-pod-no-limits-m8
      image: nginx:1.27-alpine
EOF
kubectl apply -f bad-pod-no-limits-m8.yaml

What to expect (literal, executed — Gatekeeper's message, present this time, because the object is now exactly what its Constraint declares it covers):

Error from server (Forbidden): error when creating "bad-pod-no-limits-m8.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [andes-cargo-must-have-resource-limits] container <bad-pod-no-limits-m8> is missing required resource limits: {"cpu", "memory"}
[andes-cargo-must-have-resource-limits] container <bad-pod-no-limits-m8> is missing required resource requests: {"cpu", "memory"}

And confirm, with Kyverno's logs, that it also evaluated and also blocked this same Pod — independently, with its original rule (not the autogen- one, because this time the object is, literally, a Pod):

kubectl -n kyverno logs deploy/kyverno-admission-controller --since=3m | grep "bad-pod-no-limits-m8"

What to expect (literal, executed — trimmed to the fields that matter):

... validation failed ... failed rules=["require-resource-requests-and-limits"] ... kind=Pod ... name=bad-pod-no-limits-m8 namespace=andes-cargo operation=CREATE ... policy=andes-cargo-require-resource-limits ...
... blocking admission request ... action=validate ... kind=Pod ... name=bad-pod-no-limits-m8 namespace=andes-cargo operation=CREATE ... policy=andes-cargo-require-resource-limits ...

kubectl get pod bad-pod-no-limits-m8 -n andes-cargo

What to expect:

Error from server (NotFound): pods "bad-pod-no-limits-m8" not found

There's the complete comparison: against a bare Pod, both engines reject (Gatekeeper with the message visible in kubectl, Kyverno confirmed in logs — the exact same pattern from Module 6, lesson 8). Against the real Deployment, with the same kind of violation, only Kyverno rejects, because only its rule has autogen scope over Pod controllers.

Object evaluated	Gatekeeper (`k8srequiredresources`)	Kyverno (`andes-cargo-require-resource-limits`)
Direct `Pod` (`bad-pod-no-limits-m8`)	Rejects — `match.kinds: ["Pod"]` covers this object	Rejects — original `require-resource-requests-and-limits` rule
Real `Deployment` (`andes-cargo-status-api`, via ArgoCD)	Doesn't evaluate — `match.kinds` doesn't include `Deployment`	Rejects — generated `autogen-require-resource-requests-and-limits` rule

Step 7 — Revert, and confirm full recovery

Like any real change a gate stops, the fix is a new commit — never an emergency kubectl apply:

git revert --no-edit 229cd7b
git push origin main

What to expect (literal, executed — the revert's hash is your variable value):

[main dcdd4b3] Revert "Temporarily remove resource limits from andes-cargo-status-api to debug OOM"
 1 file changed, 7 insertions(+)
To http://localhost:3000/andes-cargo/andes-cargo-k8s.git
   229cd7b..dcdd4b3  main -> main

for i in $(seq 1 20); do
  ts=$(date +%H:%M:%S)
  rev=$(kubectl get application andes-cargo-status-api -n argocd -o jsonpath='{.status.sync.revision}' | cut -c1-7)
  sync=$(kubectl get application andes-cargo-status-api -n argocd -o jsonpath='{.status.sync.status}')
  echo "$ts rev=$rev sync=$sync"
  if [ "$rev" = "dcdd4b3" ] && [ "$sync" = "Synced" ]; then break; fi
  sleep 10
done

What to expect (literal, executed):

17:28:28 rev=229cd7b sync=OutOfSync
...
17:29:19 rev=229cd7b sync=OutOfSync
17:29:29 rev=dcdd4b3 sync=Synced

kubectl delete pod bad-pod-no-limits-m8 -n andes-cargo --ignore-not-found
kubectl get k8srequiredresources
kubectl get clusterpolicy
curl -i -s --max-time 8 --resolve andes-cargo.local:80:127.0.0.1 http://andes-cargo.local/health

What to expect (literal, executed — cluster fully recovered):

NAME                                    ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
andes-cargo-must-have-resource-limits   deny                 0

NAME                                  ADMISSION   BACKGROUND   READY   AGE   MESSAGE
andes-cargo-require-resource-limits   true        true         True    60m   Ready

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 51

{"service":"andes-cargo-status-api","status":"ok"}

Analogy: two inspectors, two different lists

Picking up the factory: this lesson sent, through the correct door, an order that removed a mandatory spec from a piece. At the first quality-check station — Kyverno — the inspector has a list that covers both the individual piece and the complete blueprint that produces it: it rejected the order before the blueprint even got filed. At the second station — Gatekeeper — the inspector only has instructions to review finished pieces, never blueprints: it let the blueprint through with no comment, because it was never asked to review blueprints, only pieces. The final result — the defective piece never got manufactured — was the same, but through a different path: one inspector stopped it on paper, the other would have stopped it on the line if the first hadn't stopped it earlier.

Common mistakes

Concluding "Gatekeeper failed" or "it's weaker than Kyverno" (unfair generalization, this lesson's most important error). What happens: someone, seeing that Gatekeeper didn't react to the Deployment while Kyverno did, concludes Gatekeeper is an inferior engine. How to spot it: if your summary of this lesson is "Kyverno protects better than Gatekeeper." How to fix it: the difference isn't one of capability — it's one of configuration. This lesson's Step 6 demonstrated Gatekeeper protects exactly as well against a direct Pod. This guide's Constraint was deliberately written with match.kinds: ["Pod"] (Module 6, lesson 4) — Gatekeeper does support declaring Deployment (and other controllers) in match.kinds explicitly; this guide simply never added it, while Kyverno's ClusterPolicy's background: true activated its expanded coverage automatically, with no one requesting it field by field.

Assuming "a change the gate stops" means ArgoCD stops working (expectation about the rest of the system). What happens: someone, seeing Sync Status: OutOfSync for several minutes, assumes the entire Application stopped syncing, including the other nine resources. How to spot it: if you didn't check the individual resource table from kubectl get application ... -o yaml (status.resources) during Step 2 or 3. How to fix it: as Step 2 confirmed, only the Deployment resource stayed OutOfSync — the Namespace, the Service, the ConfigMap, the Secret, the HorizontalPodAutoscaler, the Ingress, and both NetworkPolicy stayed Synced with no issue, because none of them changed in this commit.

Trying to "fix" the problem with kubectl apply --force or editing the object directly in the cluster (misdirected urgency). What happens: someone, frustrated by seeing persistent OutOfSync, tries to force the change directly against the cluster, skipping Git entirely. How to spot it: if your first instinct in front of an admission control rejection is to look for a flag that avoids it, instead of fixing the manifest. How to fix it: an admission control rejection is never a permissions problem a flag solves — it's the policy working exactly as designed. The only legitimate fix, as Step 7 demonstrated, is a new commit that fixes the object (in this case, a git revert), never an attempt to evade the guardrail from the command line.

Exercises

Exercise 1 — Predict the result if you added Deployment to Gatekeeper's match.kinds. Without running it, predict: if you edited the andes-cargo-must-have-resource-limits Constraint so spec.match.kinds also included {"apiGroups": ["apps"], "kinds": ["Deployment"]}, and repeated this lesson's Steps 1-2, what would change in Step 3's result?

See solution

The rejection message ArgoCD captures would likely show both engines responding, with the same partial-visibility pattern Module 6, lesson 8, already documented: kubectl/ArgoCD would show only the first one in ValidatingWebhookConfiguration alphabetical order (gatekeeper-validating-webhook-configuration before kyverno-resource-validating-webhook-cfg), and Kyverno's logs would separately confirm it also evaluated and also blocked — same as this lesson's Step 6, but now against the Deployment object instead of a bare Pod. The net result (the Deployment never gets saved) wouldn't change; what would change is that now two engines, not one, would participate in the rejection.

Exercise 2 — Explain autogen without using the word "automatic." In one sentence, without using the word "automatic" or "automatically," explain what Kyverno's autogen feature this lesson discovered does.

See solution

A reasonable explanation: "When a Kyverno ClusterPolicy has background: true and a rule written for Pod, Kyverno generates, on its own, equivalent rules for the most common controllers that produce Pods — Deployment, ReplicaSet, DaemonSet, StatefulSet, Job, CronJob — applying the same criteria to the corresponding field inside each one's template, with no one having to write those additional rules by hand."

Exercise 3 — Design a test that confirms, without using logs, that Gatekeeper really didn't evaluate the Deployment. This lesson's Step 5 used kubectl logs to confirm Gatekeeper was never invoked. Without using logs, what other command, already known from Module 6, would confirm the same thing?

See solution

kubectl get k8srequiredresources andes-cargo-must-have-resource-limits — the TOTAL-VIOLATIONS column reflects what gatekeeper-audit finds when re-evaluating the cluster's current state in the background (Module 6, lesson 4). If Gatekeeper considered the Deployment within its scope, and the Deployment had at some point been left with no resources (something that never happened, because Kyverno blocked it first), TOTAL-VIOLATIONS would report it. Since the Constraint never included Deployment in its match.kinds, that number stays at 0 regardless of what happens to the Deployment — an indirect but valid confirmation that this object type is outside its scope.

Summary and next step

This lesson pushed, through the same GitOps path as lesson 3, a real change that violates Module 6's resource policy — and confirmed, with literal evidence, that the gate stops it: the live Deployment never changed, no new Pod got created, and Sync Status stayed OutOfSync until a new commit (a git revert) fixed the problem. This lesson's unplanned finding — Kyverno, via its autogen mechanism, caught the Deployment directly, while Gatekeeper, with the Constraint configured the way Module 6 left it, only covers Pod objects — is a real lesson about admission-control configuration in production: two engines that solve the same problem can, with no one planning it, end up covering slightly different scopes, and the only way to know for sure is to test against the real object, not just against the simplest test case.

Before moving on you should be able to: reproduce this complete walkthrough in your own lab, including the revert; explain why Kyverno caught the Deployment and Gatekeeper didn't, without using the word "better" or "worse"; and design a test, with or without logs, that confirms any Constraint/ClusterPolicy's real scope before trusting it in production.

Next lesson: what this guide left representative. There, the complete honesty about EKS — the only thing in this entire guide that never ran against a real cluster — gets gathered in one place, before closing the capstone with the map of the rest of the ecosystem.

Resources

Kyverno — Auto-Gen Rules for Pod Controllers — official documentation for the autogen- mechanism this lesson discovered in production.
Gatekeeper — Constraints, match — official reference for spec.match.kinds, the exact field that explains why Gatekeeper didn't evaluate the Deployment.
Argo CD — Sync Status and Argo CD — Diffing — official documentation for OutOfSync and how ArgoCD reports a real application failure.
kubernetes-and-eks-in-production-guide (NIEVA), Module 6, lessons 4, 6, and 8 — the origin of the Constraint/ClusterPolicy and the original "side by side" test this lesson extends.

← Previous3. End-to-end walkthrough: a change that crosses the whole gate Next →5. What this guide left representative, final honesty

4. End-to-end walkthrough: a change the gate stops

Description

Connection to the module

Step 0 — The scenario: a real reason, a dangerous change

Step 1 — The change: remove the entire resources block

Step 2 — ArgoCD detects it, and gets stuck: OutOfSync, not Synced

Step 3 — The literal message: Kyverno blocked the Deployment, before it got saved

Step 4 — Confirm: the real Deployment never changed

Step 5 — The real finding: Gatekeeper never got to evaluate this attempt

Step 6 — Reproduce the "side by side" comparison Module 6 showed: a bare Pod, with no Deployment in the way

Step 7 — Revert, and confirm full recovery

Analogy: two inspectors, two different lists

Common mistakes

Exercises

Summary and next step

Resources

Step 1 — The change: remove the entire `resources` block

Step 2 — ArgoCD detects it, and gets stuck: `OutOfSync`, not `Synced`

Step 3 — The literal message: Kyverno blocked the `Deployment`, before it got saved

Step 4 — Confirm: the real `Deployment` never changed

Step 6 — Reproduce the "side by side" comparison Module 6 showed: a bare Pod, with no `Deployment` in the way