Module 6: Runtime Security Admission Control And Image Scanning
8. Project: Andes Cargo's runtime guardrails
Description
This project closes the module with two pieces the previous lessons deliberately kept separate. First, it reactivates Gatekeeper (which lesson 6 put in dryrun to isolate Kyverno's test) and leaves both engines active at once over andes-cargo — something no earlier exercise tested, and that this project documents with real evidence, with nothing guessed. Second, it builds a real gate — trivy image as a required step before kind load docker-image — and runs it against Andes Cargo's own image, with the honest result that image produces today: it fails.
Connection to the module
Everything that follows uses exclusively pieces already built in this module: Gatekeeper's Constraint (lesson 4), Kyverno's ClusterPolicy (lesson 6), and trivy image's real findings (lesson 7) — this project introduces no new concept, it only combines them and observes what happens when they coexist.
Step 1 — Reactivate Gatekeeper: both engines, active at once
kubectl patch k8srequiredresources andes-cargo-must-have-resource-limits \
--type merge -p '{"spec":{"enforcementAction":"deny"}}'
kubectl get k8srequiredresources andes-cargo-must-have-resource-limits
kubectl get clusterpolicy andes-cargo-require-resource-limitsWhat to expect:
k8srequiredresources.constraints.gatekeeper.sh/andes-cargo-must-have-resource-limits patched
NAME ENFORCEMENT-ACTION TOTAL-VIOLATIONS
andes-cargo-must-have-resource-limits deny 0
NAME ADMISSION BACKGROUND READY AGE MESSAGE
andes-cargo-require-resource-limits true true True ... ReadyFrom this moment on, any Pod trying to get created in andes-cargo goes through two different ValidatingWebhookConfigurations in the same admission phase — Gatekeeper's (gatekeeper-validating-webhook-configuration) and Kyverno's (kyverno-resource-validating-webhook-cfg), both evaluating essentially the same rule, with the same requirement over resources.limits/requests.
kubectl get validatingwebhookconfigurationsWhat to expect:
NAME WEBHOOKS AGE
gatekeeper-validating-webhook-configuration 2 ...
ingress-nginx-admission 1 ...
kyverno-cel-exception-validating-webhook-cfg 1 ...
kyverno-cleanup-validating-webhook-cfg 1 ...
kyverno-exception-validating-webhook-cfg 1 ...
kyverno-global-context-validating-webhook-cfg 1 ...
kyverno-policy-validating-webhook-cfg 1 ...
kyverno-resource-validating-webhook-cfg 1 ...
kyverno-ttl-validating-webhook-cfg 1 ...Notice the order: gatekeeper-validating-webhook-configuration shows up before any kyverno-*, alphabetically (g before i, before k). Keep this observation in mind — you're going to need it to explain Step 2's result.
Step 2 — A Pod that violates it, with both engines active: tested twice
echo "=== attempt 1 ==="
kubectl apply -f bad-pod-no-limits.yaml
echo
echo "=== attempt 2 ==="
kubectl apply -f bad-pod-no-limits.yamlWhat to expect (literal — both attempts, run one after the other, with exactly the same result):
=== attempt 1 ===
Error from server (Forbidden): error when creating "bad-pod-no-limits.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [andes-cargo-must-have-resource-limits] container <bad-pod-no-limits> is missing required resource limits: {"cpu", "memory"}
[andes-cargo-must-have-resource-limits] container <bad-pod-no-limits> is missing required resource requests: {"cpu", "memory"}
=== attempt 2 ===
Error from server (Forbidden): error when creating "bad-pod-no-limits.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [andes-cargo-must-have-resource-limits] container <bad-pod-no-limits> is missing required resource limits: {"cpu", "memory"}
[andes-cargo-must-have-resource-limits] container <bad-pod-no-limits> is missing required resource requests: {"cpu", "memory"}The Pod was rejected both times, consistently — that confirms the guardrail works. But notice something more specific: in both attempts, kubectl only shows you Gatekeeper's message (validation.gatekeeper.sh). Not once did Kyverno's message (validate.kyverno.svc-fail), the one you saw in lesson 6, show up. The honest question this project has to answer: does this mean Kyverno never got to evaluate the Pod, or that it did evaluate it and its response simply wasn't shown?
Step 3 — The real test: did Kyverno evaluate the Pod, even though its message didn't show up?
Kubernetes invokes every ValidatingWebhookConfiguration that matches an object — it doesn't stop at the first one that responds. But when several webhooks deny the same request, kube-apiserver isn't required to show you every one's messages; in this cluster's practice, it showed only the first one it processed. kyverno-admission-controller's own logs, however, don't depend on what kube-apiserver showed your terminal — they record what Kyverno itself did:
kubectl -n kyverno logs deploy/kyverno-admission-controller --since=5m | \
grep "bad-pod-no-limits"What to expect (literal — trimmed to the fields that matter; Kyverno's full log includes more metadata per line):
... validation failed ... failed rules=["require-resource-requests-and-limits"] ... kind=Pod ... name=bad-pod-no-limits namespace=andes-cargo operation=CREATE ... policy=andes-cargo-require-resource-limits ...
... blocking admission request ... action=validate ... kind=Pod ... name=bad-pod-no-limits namespace=andes-cargo operation=CREATE ... policy=andes-cargo-require-resource-limits ...This is the direct proof: it shows up twice — once per Step 2 attempt, with different timestamps — each time with validation failed followed by blocking admission request. Kyverno did evaluate the Pod, did find the violation, and did decide to block it, completely independently of Gatekeeper — its decision simply never made it into your terminal, because the message kubectl printed was Gatekeeper's.
The real finding: how Gatekeeper and Kyverno interact when they coexist
kubectl apply -f bad-pod-no-limits.yaml
│
▼
┌───────────────────┐
│ kube-apiserver │
│ phase 3: admission │
└─────────┬─────────┘
│ invokes EVERY ValidatingWebhookConfiguration
│ that matches (doesn't stop at the first one)
┌─────────┴─────────┐
▼ ▼
gatekeeper-validating- kyverno-resource-
webhook-configuration validating-webhook-cfg
(evaluates, denies) (evaluates, denies —
│ confirmed in logs)
│ │
└─────────┬─────────┘
│ BOTH responded allowed: false
▼
kube-apiserver rejects the request
(shows ONLY Gatekeeper's message,
the first one in the alphabetical order
of ValidatingWebhookConfiguration names
observed in this cluster — "gatekeeper-..."
before "kyverno-...")Three honest conclusions, verified twice in this lab, none invented:
- There was no functional conflict between the two engines. Both evaluated the same object, both reached the same conclusion (reject), and the net result for the cluster — the Pod never reached
etcd— was identical to either engine acting alone. "Clashing" in the sense of one engine undoing or contradicting the other did not happen. - There was a loss of information for the user.
kubectlonly showed one message, Gatekeeper's — without checking Step 3's logs, it would have been impossible to confirm, from the terminal, that Kyverno also evaluated and also rejected. In a real cluster with two active engines and overlapping policies, this is exactly the kind of operational ambiguity lesson 5 of this module named as a real reason most teams end up standardizing on a single engine — not a technical limitation preventing them from coexisting. - The observed order (Gatekeeper before Kyverno) matched, in this cluster, the alphabetical order of
ValidatingWebhookConfigurationnames (gatekeeper-validating-webhook-configurationbeforekyverno-resource-validating-webhook-cfg) — but Kubernetes' official documentation does not guarantee any specific invocation order between differentValidatingWebhookConfigurations. This guide reports what it observed, twice, consistently, on this specific cluster — it doesn't present it as a rule guaranteed by the Kubernetes API for any cluster.
Step 4 — The image gate: trivy image before kind load docker-image
This project's second half turns lesson 7's finding into a real operational guardrail: a script that scans an image with trivy image before loading it into the cluster, and aborts if it finds CRITICAL findings.
mkdir -p scripts
cat > scripts/scan-and-load.sh << 'EOF'
#!/usr/bin/env bash
# scan-and-load.sh — trivy image gate before kind load docker-image
# Usage: ./scan-and-load.sh <image:tag>
set -euo pipefail
IMAGE="$1"
CLUSTER="andes-cargo-cluster"
echo "==> scanning ${IMAGE} for CRITICAL vulnerabilities before loading into ${CLUSTER}"
if trivy image --exit-code 1 --severity CRITICAL --quiet "${IMAGE}"; then
echo "==> gate PASSED — no CRITICAL findings, loading ${IMAGE} into ${CLUSTER}"
kind load docker-image "${IMAGE}" --name "${CLUSTER}"
echo "==> ${IMAGE} loaded"
else
echo "==> gate FAILED — ${IMAGE} has CRITICAL findings, NOT loaded into ${CLUSTER}"
exit 1
fi
EOF
chmod +x scripts/scan-and-load.shThe script invents no new Trivy capability — it chains two commands you already know (trivy image --exit-code 1 --severity CRITICAL, which lesson 7 didn't use yet, and kind load docker-image, from Module 1) with simple shell logic: if the scan fails (there's at least one CRITICAL), the script never reaches the kind load line.
Run 1 — against andes-cargo-status-api:latest: the gate really fails
./scripts/scan-and-load.sh andes-cargo-status-api:latest
echo "exit code: $?"What to expect (literal — trimmed to the relevant rows; Trivy's full report is the same one lesson 7 already showed in full):
==> scanning andes-cargo-status-api:latest for CRITICAL vulnerabilities before loading into andes-cargo-cluster
andes-cargo-status-api:latest (debian 13.6)
===========================================
Total: 4 (CRITICAL: 4)
┌───────────┬────────────────┬──────────┬──────────────┬───────────────────┬───────────────┐
│ Library │ Vulnerability │ Severity │ Status │ Installed Version │ Fixed Version │
├───────────┼────────────────┼──────────┼──────────────┼───────────────────┼───────────────┤
│ perl-base │ CVE-2026-13221 │ CRITICAL │ affected │ 5.40.1-6 │ │
│ │ CVE-2026-42496 │ │ fix_deferred │ │ │
│ │ CVE-2026-57433 │ │ affected │ │ │
│ │ CVE-2026-8376 │ │ │ │ │
└───────────┴────────────────┴──────────┴──────────────┴───────────────────┴───────────────┘
==> gate FAILED — andes-cargo-status-api:latest has CRITICAL findings, NOT loaded into andes-cargo-cluster
exit code: 1This is uncomfortable honesty on purpose, not a lab accident: Andes Cargo's own production image, running in andes-cargo-cluster since Module 1, would not pass this gate today if you applied it strictly — the same four perl-base CRITICALs lesson 7 found, with no Fixed Version available yet. A real team, facing this exact result, has three honest paths — the same pattern cloud-security-and-guardrails-guide already established: fix (doesn't apply here, no patch available yet), suppress with documented justification (mark these four specific CVEs as accepted, with a review date, in a .trivyignore file — a policy decision, not a technical one), or explicitly accept the risk and adjust the gate's threshold (for example, block only if a new CRITICAL appears since the last review, not any CRITICAL with no exceptions). This project doesn't make that decision for you — it leaves it documented, with the gate working exactly as it should: stopping, with real evidence, before anyone has to discover it in production.
Run 2 — against a clean image: the gate passes, and really loads
docker pull hello-world:latest
./scripts/scan-and-load.sh hello-world:latest
echo "exit code: $?"What to expect:
==> scanning hello-world:latest for CRITICAL vulnerabilities before loading into andes-cargo-cluster
==> gate PASSED — no CRITICAL findings, loading hello-world:latest into andes-cargo-cluster
Image: "hello-world:latest" with ID "sha256:eb84fdc6..." not yet present on node "andes-cargo-cluster-worker", loading...
Image: "hello-world:latest" with ID "sha256:eb84fdc6..." not yet present on node "andes-cargo-cluster-control-plane", loading...
Image: "hello-world:latest" with ID "sha256:eb84fdc6..." not yet present on node "andes-cargo-cluster-worker2", loading...
==> hello-world:latest loaded
exit code: 0hello-world:latest has no Debian/Alpine operating system behind it — it's, deliberately, the most minimal image that exists on Docker Hub — so Trivy finds no package to scan, and the gate passes clean. The sha256 is variable (it depends on the exact version Docker Hub serves); the pattern of the three "not yet present... loading..." lines — one per cluster node — is fixed, the same mechanism Module 1 already used for andes-cargo-status-api:latest.
Additional evidence: what the real cluster already complies with
Before closing the project, it's worth confirming something the two previous sections took for granted: the real andes-cargo-status-api Deployment — the one running in production since Module 1, with Module 3's limits — still passes both policies, with no change from you in this module.
kubectl get policyreport -n andes-cargoWhat to expect:
NAME KIND NAME PASS FAIL WARN ERROR SKIP AGE
417220d1-3f46-476c-a6f8-ac1a5e501853 Pod andes-cargo-status-api-548966dd97-xx6zs 1 0 0 0 0 ...
6097b68c-4f8f-41c7-8875-31aab943cb5d Pod andes-cargo-status-api-548966dd97-gsgx8 1 0 0 0 0 ...
c3058c00-f825-43c8-b68c-e03e608840fd Deployment andes-cargo-status-api 1 0 0 0 0 ...
debf13b0-c282-446c-a1b8-184a60c47e92 ReplicaSet andes-cargo-status-api-548966dd97 1 0 0 0 0 ...PASS: 1, FAIL: 0 on all four rows — it's Kyverno (kyverno-reports-controller, in the background) confirming the same thing Gatekeeper's TOTAL-VIOLATIONS: 0 already showed since lesson 4: this module's guardrail didn't have to fix anything because Module 3 already did it right. Each NAME's hash suffixes are variable.
Module 6's final checklist
| Piece | Engine/tool | Status |
|---|---|---|
| Rejecting a Pod with no resource limits | Gatekeeper v3.23.0, isolated | Executed (lesson 4) — literal message captured |
| Rejecting a Pod with no resource limits | Kyverno v1.18.2, isolated | Executed (lesson 6) — literal message captured |
| Rejecting a Pod with no resource limits | Gatekeeper + Kyverno, both active | Executed, twice (this project) — Gatekeeper's message in kubectl, Kyverno's evaluation confirmed in logs |
| Admitting a Pod with resource limits | Gatekeeper and Kyverno, each separately | Executed (lessons 4 and 6) — 1/1 Running in both cases |
| Image vulnerability scan | trivy image | Executed (lesson 7) — 184 real findings, 4 CRITICAL with no patch available |
Image gate before kind load docker-image | trivy image --exit-code 1 --severity CRITICAL + scripts/scan-and-load.sh | Executed, twice — real failure against andes-cargo-status-api:latest, real pass against hello-world:latest |
Confirmation the real Deployment complies with both policies | Kyverno's PolicyReport + Gatekeeper's TOTAL-VIOLATIONS | Executed — PASS: 1, FAIL: 0 on all four namespace rows |
Cleanup: leave the cluster in a consistent state
kubectl delete pod bad-pod-no-limits -n andes-cargo --ignore-not-found
docker rmi hello-world:latest 2>/dev/null || trueThe first command confirms no test Pod was left running (Step 2's two attempts were both rejected, so in theory there should be nothing to delete — --ignore-not-found avoids an error if that's the case). The second is optional, just to avoid leaving an unused test image on your local Docker.
Common mistakes
Interpreting Step 3's finding as "Gatekeeper and Kyverno are in conflict" (imprecise language). What happens: someone, seeing only one message show up in kubectl while both engines evaluated, concludes they "clash" in the sense of breaking or contradicting each other. How to spot it: if your summary of this project is "Gatekeeper and Kyverno aren't compatible with each other." How to fix it: revisit this lesson's "The real finding" section — both engines reached the same conclusion, completely independently, and the net result (the Pod never existed) was identical. All that "gets lost" is visibility for the human reading the terminal, not the guardrail's correctness — the cluster is protected just as well with one engine active or both.
Assuming this project's --severity CRITICAL gate is the only valid way to define a threshold (unnecessary rigidity). What happens: someone copies scan-and-load.sh literally and assumes "block on any CRITICAL, with no exceptions" is the only reasonable policy for any team. How to spot it: if your reaction, after seeing Run 1 fail against Andes Cargo's own image, is "then nothing could ever be deployed." How to fix it: the threshold (--severity CRITICAL, no exceptions) is a design decision specific to this project, chosen on purpose so you'd see a real result, not an artificial one. A real team, facing the same result, would normally adjust the gate — excluding specific CVEs already reviewed and accepted, with .trivyignore and a review date, or blocking only findings new since the last audit — instead of blocking indefinitely a service that otherwise works fine.
Not checking Step 3's Kyverno logs, and being left unsure whether it really evaluated the Pod (incomplete verification). What happens: someone reads Step 2, sees only Gatekeeper's message, and moves on without running Step 3 — left, without knowing it, with no evidence confirming or refuting whether Kyverno participated. How to spot it: if you finished this project without having seen, with your own eyes, the validation failed/blocking admission request lines in kyverno-admission-controller's logs. How to fix it: run Step 3's command before considering this project closed — it's the only way to turn "probably both evaluated it" into "confirmed, with timestamps, that both evaluated it."
Exercises
Exercise 1 — Design a test that confirms the observed order with a hypothetical third engine. If you installed a hypothetical third admission-control engine, with a ValidatingWebhookConfiguration named aaa-first-policy-engine, what would you expect to see in the rejection message of a Pod all three policies reject at once, based on the pattern you observed in this project?
See solution
Based on the observed pattern (Gatekeeper, named gatekeeper-..., consistently appeared before Kyverno, named kyverno-..., in both of Step 2's attempts), it would be reasonable to predict that aaa-first-policy-engine — alphabetically before both gatekeeper- and kyverno- — would show up as the message displayed in kubectl. But the complete, honest answer has to include this lesson's warning: this project observed that order consistently on this specific cluster, with no guarantee from Kubernetes' official documentation that this behavior holds for any cluster or any kube-apiserver version — the only way to confirm it with certainty would be, as in Step 3, checking each engine's logs separately, not relying solely on which message kubectl decides to show.
Exercise 2 — Explain the gate's threshold decision to a colleague who asks "why don't we also block HIGH?" A teammate, seeing that scan-and-load.sh only blocks on CRITICAL, asks why it doesn't also include HIGH — after all, lesson 7 found 21 real HIGH findings. What would you answer them?
See solution
A reasonable answer: "It's a threshold decision, not a technical limitation of the script — changing --severity CRITICAL to --severity CRITICAL,HIGH is a single flag. The reason to start with CRITICAL only is to show ourselves, with this very lab, how strict we can get without indefinitely blocking our own production image: we already saw that even with the most permissive threshold (CRITICAL only), the current image fails the gate. Raising it to HIGH would add 21 more findings to that list, most in the base operating system, with no change on our part resolving them immediately. Before raising the threshold, we'd need to review those 21 one by one and decide which ones we accept — the same fix/suppress/accept process we already used with the CRITICALs, applied to a longer list."
Exercise 3 — Predict what would happen if you deleted Gatekeeper's Constraint but left Kyverno's ClusterPolicy active, and repeated Step 2. Without running anything, predict: if you ran kubectl delete k8srequiredresources andes-cargo-must-have-resource-limits and then repeated the attempt to create bad-pod-no-limits.yaml, what message would you expect to see, and how would it differ from this project's Step 2?
See solution
The Pod would still get rejected — Kyverno, with its ClusterPolicy still active, is completely independent of whether Gatekeeper's Constraint exists or not — but the message kubectl would show would, this time, be Kyverno's (admission webhook "validate.kyverno.svc-fail" denied the request..., the exact same text you saw in lesson 6), because there would no longer be any other webhook competing to show up first. This experiment, if you actually run it, is the most direct way to confirm the two engines act genuinely independently: removing one changes nothing about the protection the other keeps offering.
Summary and next step
This project activated Gatekeeper and Kyverno at once over andes-cargo, and documented, with real evidence (two identical attempts, plus Kyverno's logs), that both engines evaluate every object independently and reach the same conclusion — with no functional conflict, even though kubectl only shows one of the two messages. It also built a real trivy image gate before kind load docker-image, and ran it twice: it failed, honestly, against Andes Cargo's own production image (four CRITICALs with no patch available, inherited from perl-base), and it passed, actually loading, against an image with no operating system behind it. andes-cargo-cluster closes this module with a real, doubled gatekeeper at its single entry door — and with the complete honesty of what that gatekeeper would do if you put it to evaluate its own workload today.
Before moving on you should be able to: explain, without using the word "conflict," what happens when two ValidatingWebhookConfigurations evaluate the same object; reproduce the trivy image gate before any future kind load docker-image; and name the three honest paths facing a CRITICAL finding with no patch available (fix, suppress with justification, or explicitly accept the risk).
Next module: what's specific to EKS in production. Module 7 takes everything built on kind — Pods, Services, GitOps, and now these runtime guardrails — and confronts the question no earlier module could answer with a local cluster: what really changes when the control plane is no longer administered by you, but by AWS.
Resources
- Kubernetes — Dynamic Admission Control — official behavior for invoking multiple
ValidatingWebhookConfigurations over the same object. - Gatekeeper — Violations and Kyverno — Policy Reports — the two evidence sources this project cross-checked to confirm both engines evaluated independently.
- Trivy — Container Image — reference for
--exit-code/--severity, the two flags that turn an informational scan into a real gate. - kind — Load a local image into your cluster — official documentation for
kind load docker-image, the command this project puts behind the gate. cloud-security-and-guardrails-guide(NIEVA), Module 5, lesson 6 — the "fix, suppress, or accept" pattern this project picks back up against the fourCRITICALs with no patch available.