Module 6: Runtime Security Admission Control And Image Scanning

6. Hands-on: the same policy, in Kyverno

Next →

Description

This lesson actually installs Kyverno v1.18.2 against andes-cargo-cluster — the same cluster where Gatekeeper has kept running since lesson 4 — applies lesson 5's ClusterPolicy, and repeats exactly the same test: a Pod that violates it, a Pod that complies. Before installing anything, this lesson makes an explicit operational decision — and explains it — so Kyverno's test doesn't get confused by the Gatekeeper Constraint already active.

Connection to the module

Gatekeeper and Kyverno, if both have enforcementAction/validationFailureAction in blocking mode at once, and both match over the same namespace with the same rule, are going to reject the same Pod both of them, in parallel. That's exactly what this module's lesson 8 (the project) is going to test on purpose, with complete evidence of how they interact. This lesson, on the other hand, needs an isolated test of Kyverno — so, before installing anything, the first step is temporarily turning off enforcement of Gatekeeper's Constraint, without deleting it.

Step 1 — Isolate the test: put Gatekeeper's Constraint in dryrun

kubectl patch k8srequiredresources andes-cargo-must-have-resource-limits \
  --type merge -p '{"spec":{"enforcementAction":"dryrun"}}'
kubectl get k8srequiredresources andes-cargo-must-have-resource-limits

What to expect:

k8srequiredresources.constraints.gatekeeper.sh/andes-cargo-must-have-resource-limits patched
NAME                                    ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
andes-cargo-must-have-resource-limits   dryrun               0

dryrun means Gatekeeper keeps evaluating every object against the rule — you're going to be able to confirm it with TOTAL-VIOLATIONS if you apply something that violates it — but it no longer blocks anything: any Pod, whether it complies or not, reaches etcd either way. This is the exact mode lesson 4 named and didn't use yet. The reason for this step, said plainly: if you left Gatekeeper on deny while testing Kyverno, the test Pod with no limits would get rejected by the wrong engine — Gatekeeper would respond first, and you'd never know whether Kyverno's ClusterPolicy, by itself, would have rejected it too. Isolating the test is what lets you attribute the result to the correct engine.

Step 2 — Install Kyverno v1.18.2

curl -sL -o kyverno-install.yaml \
  https://github.com/kyverno/kyverno/releases/download/v1.18.2/install.yaml
kubectl create -f kyverno-install.yaml

Notice this command uses kubectl create, not kubectl apply — unlike Gatekeeper's manifest. The reason is purely technical: Kyverno's manifest includes CRD definitions with extensive schemas (complete openAPIV3Schemas, to validate every field of a ClusterPolicy), and kubectl apply saves a complete copy of the previous object as an annotation so it can compute diffs on the next apply — with schemas this size, that annotation can exceed the size limit Kubernetes allows for a single annotation. kubectl create avoids the problem entirely: it creates the objects directly, with no "last applied configuration" history saved. It's the same installation, documented this way by the Kyverno project itself.

What to expect (fixed — the official manifest doesn't change the list of resources it creates):

namespace/kyverno created
serviceaccount/kyverno-admission-controller created
configmap/kyverno created
customresourcedefinition.apiextensions.k8s.io/clusterpolicies.kyverno.io created
customresourcedefinition.apiextensions.k8s.io/policies.kyverno.io created
customresourcedefinition.apiextensions.k8s.io/policyreports.wgpolicyk8s.io created
...
clusterrole.rbac.authorization.k8s.io/kyverno:admission-controller created
service/kyverno-svc created
deployment.apps/kyverno-admission-controller created
deployment.apps/kyverno-background-controller created
deployment.apps/kyverno-cleanup-controller created
deployment.apps/kyverno-reports-controller created

Four Deployments, not one — a real architectural difference from Gatekeeper, which installed two (gatekeeper-controller-manager and gatekeeper-audit): kyverno-admission-controller (the one that responds to the ValidatingWebhookConfiguration, the direct equivalent of gatekeeper-controller-manager), kyverno-background-controller (re-evaluates existing objects, the equivalent of gatekeeper-audit), kyverno-cleanup-controller (deletes resources per scheduled cleanup policies, with no equivalent in this module), and kyverno-reports-controller (generates the PolicyReports you're going to see later in this module).

Wait for the admission controller to be ready:

kubectl -n kyverno rollout status deployment/kyverno-admission-controller --timeout=180s
kubectl -n kyverno get pods

What to expect:

deployment "kyverno-admission-controller" successfully rolled out
NAME                                             READY   STATUS    RESTARTS   AGE
kyverno-admission-controller-656b594944-vzv79    1/1     Running   0          30s
kyverno-background-controller-59bd999b84-rc84p   1/1     Running   0          30s
kyverno-cleanup-controller-657f9dd6d5-pt5lx      1/1     Running   0          30s
kyverno-reports-controller-746c796cf8-l7sc4      1/1     Running   0          30s

Hash suffixes and AGE are variable, as always.

Step 3 — The ClusterPolicy: the equivalent policy, in YAML

mkdir -p kyverno
cat > kyverno/policy-andes-cargo-require-resources.yaml << 'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: andes-cargo-require-resource-limits
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: require-resource-requests-and-limits
      match:
        any:
          - resources:
              kinds:
                - Pod
              namespaces:
                - andes-cargo
      validate:
        message: "every container must set resources.requests and resources.limits for cpu and memory"
        pattern:
          spec:
            containers:
              - resources:
                  requests:
                    cpu: "?*"
                    memory: "?*"
                  limits:
                    cpu: "?*"
                    memory: "?*"
EOF

kubectl apply -f kyverno/policy-andes-cargo-require-resources.yaml
kubectl get clusterpolicy andes-cargo-require-resource-limits

What to expect:

clusterpolicy.kyverno.io/andes-cargo-require-resource-limits created
NAME                                  ADMISSION   BACKGROUND   READY   AGE   MESSAGE
andes-cargo-require-resource-limits   true        true         True    3s    Ready

ADMISSION: true confirms this policy evaluates in real time, at the moment of any new Pod's kubectl apply — the same instant lesson 2's phase 3 described. BACKGROUND: true (inherited from spec.background: true in the YAML) confirms that, in addition, kyverno-background-controller is going to re-evaluate already-existing objects, the equivalent of gatekeeper-audit. READY: True confirms Kyverno finished registering the policy with no validation error against its own schema.

Step 4 — The Pod that violates it: rejected (with Kyverno as the only active engine)

kubectl apply -f bad-pod-no-limits.yaml

What to expect (literal — Kyverno's real rejection message, with Gatekeeper in dryrun and therefore not interfering):

Error from server: error when creating "bad-pod-no-limits.yaml": admission webhook "validate.kyverno.svc-fail" denied the request:

resource Pod/andes-cargo/bad-pod-no-limits was blocked due to the following policies

andes-cargo-require-resource-limits:
  require-resource-requests-and-limits: 'validation error: every container must set
    resources.requests and resources.limits for cpu and memory. rule require-resource-requests-and-limits
    failed at path /spec/containers/0/resources/limits/'

Compare this message with lesson 4's from Gatekeeper, and notice the real formatting differences:

  • admission webhook "validate.kyverno.svc-fail" — a webhook name completely different from lesson 4's "validation.gatekeeper.sh", even though both fulfill exactly the same role within phase 3 of the lifecycle.
  • 'validation error: every container must set resources.requests and resources.limits...' — this is, literally, the text you put in spec.rules[].validate.message in Step 3's YAML — unlike Gatekeeper's message (dynamically generated by the sprintf function inside Rego), Kyverno's message is, almost always, the fixed text you wrote yourself.
  • failed at path /spec/containers/0/resources/limits/ — Kyverno gives you the exact path, inside the object's YAML, where the pattern didn't match — information Gatekeeper, in this example, gave you differently (which fields are missing, {"cpu", "memory"}), not exactly where in the structure.

And, exactly like with Gatekeeper: kubectl get pods -n andes-cargo | grep bad-pod shows nothing. The object never existed.

Step 5 — The Pod that complies with it: admitted

kubectl apply -f good-pod-with-limits.yaml
kubectl get pod good-pod-with-limits -n andes-cargo

What to expect:

pod/good-pod-with-limits created
NAME                   READY   STATUS    RESTARTS   AGE
good-pod-with-limits   1/1     Running   0           5s

The exact same YAML from lesson 4, now evaluated by a completely different engine, with the same result: 1/1 Running. Clean up the Pod before continuing:

kubectl delete pod good-pod-with-limits -n andes-cargo

Side-by-side comparison: both engines, the same result

Gatekeeper (lesson 4)Kyverno (this lesson)
Objects needed2 (ConstraintTemplate + Constraint)1 (ClusterPolicy)
Rule languageRego (violation[...] { ... })YAML (pattern, "?*" operator)
Webhook name in the rejectionvalidation.gatekeeper.shvalidate.kyverno.svc-fail
Block/audit switchspec.enforcementAction (deny/dryrun/warn)spec.validationFailureAction (Enforce/Audit)
Rejection messageDynamically generated by Rego logicThe fixed text of validate.message, almost always
Admission Deploymentgatekeeper-controller-manager (3 replicas)kyverno-admission-controller
Background re-evaluation Deploymentgatekeeper-auditkyverno-background-controller
Pod with no limits (bad-pod-no-limits)Rejected, never reaches etcdRejected, never reaches etcd
Pod with limits (good-pod-with-limits)Admitted, 1/1 RunningAdmitted, 1/1 Running

That table's last row is this lesson's real conclusion: for the result that matters to the cluster's user, the two engines are indistinguishable. The difference lives entirely in how each team, on the side of whoever writes the policy, prefers to express it — not in how well they protect the cluster.

Common mistakes

Forgetting Step 1 and getting confused about which engine rejected the Pod (test isolation). What happens: someone jumps straight to Step 4 without putting Gatekeeper in dryrun, and the rejection message shows validation.gatekeeper.sh, not validate.kyverno.svc-fail — it looks like Kyverno "did nothing." How to spot it: if the webhook name in your error message doesn't match what this lesson shows. How to fix it: confirm with kubectl get k8srequiredresources andes-cargo-must-have-resource-limits that ENFORCEMENT-ACTION says dryrun, not deny. If it's still on deny, repeat Step 1's kubectl patch — without that step, Gatekeeper (already active since lesson 4) is going to respond first, and the message you see is going to be its own, not Kyverno's. This isn't a Kyverno configuration mistake; it's exactly the interaction phenomenon between two engines this module's lesson 8 documents on purpose.

Using kubectl apply instead of kubectl create to install Kyverno, and running into an annotation-size error (installation). What happens: kubectl apply -f kyverno-install.yaml fails with a message about metadata.annotations: Too long: must have at most 262144 bytes, or a similar error about last-applied-configuration's size. How to spot it: the error explicitly mentions an annotation's size limit. How to fix it: use kubectl create -f kyverno-install.yaml, as this lesson's Step 2 does — it completely avoids the annotation mechanism that causes the problem. If you already installed Kyverno with apply and it failed partway through, run kubectl delete -f kyverno-install.yaml before retrying with create, to avoid leaving partial resources.

Confusing background: true with "this policy only runs in the background, never at admission" (incomplete reading of the field). What happens: someone reads spec.background: true in Step 3's YAML and assumes the policy blocks nothing in real time. How to spot it: if your expectation, before Step 4, was that bad-pod-no-limits.yaml would get created without issue. How to fix it: background: true is additive, not exclusive — it means "in addition to evaluating at the moment of apply (any validate rule's default behavior), also re-evaluate objects that already exist, periodically, in the background." The ADMISSION: true you saw in Step 3's output is the field that confirms real-time evaluation stays active; background doesn't replace it, it complements it.

Exercises

Exercise 1 — Predict the READY/MESSAGE of a ClusterPolicy with malformed YAML. Without running anything, predict: if Step 3's pattern had a YAML indentation error (for example, limits at the same level as requests instead of nested inside resources), what would you expect to see in the READY column of kubectl get clusterpolicy?

See solution

It would depend on the exact type of error: syntactically invalid YAML (indentation broken to the point it doesn't even parse as valid YAML) would make kubectl apply fail immediately, never getting to create the ClusterPolicy at all — the error would show up in your own terminal, not in the object's state. Syntactically valid but semantically incorrect YAML for Kyverno's schema (for example, a field with the wrong type) would create the object, but would show READY: False, with a MESSAGE describing the specific validation issue — the same pattern you already saw with k8srequiredresources in Gatekeeper when its openAPIV3Schema rejects a malformed Constraint.

Exercise 2 — Explain, in one sentence, why Kyverno's rejection message includes a path (/spec/containers/0/resources/limits/) and Gatekeeper's, in this example, doesn't. Without repeating this lesson's text, what causes that difference?

See solution

It's due to how each engine generates its message internally, not a limitation of one over the other: Kyverno's pattern is, literally, a structural comparison against the object's YAML tree — when the comparison fails, Kyverno already knows exactly at which node of the tree the mismatch occurred, and reports it as a path. Gatekeeper's Rego rule, in this specific ConstraintTemplate, built its message with sprintf from a set (missing) of missing field names — a decision about how that particular rule was written, not a language limitation: a different Rego rule could perfectly well include the path too, if whoever wrote it decided to build the message that way.

Exercise 3 — Design the check you'd run before reactivating Gatekeeper to deny. Before lesson 8 reactivates Gatekeeper's Constraint (switching it from dryrun back to deny), what command would you run to confirm that, while it was in dryrun, Gatekeeper did evaluate the Pod with no limits (even though it didn't block it), as evidence that dryrun audits without blocking?

See solution

kubectl get k8srequiredresources andes-cargo-must-have-resource-limits — if at any point during this lesson a Pod with no limits had ended up running in andes-cargo (for example, if Kyverno hadn't blocked it and you hadn't cleaned it up), that Constraint's TOTAL-VIOLATIONS column would show a number greater than 0, confirming gatekeeper-audit did register the violation in the background, even though dryrun never prevented the object from getting to exist. In this specific lab, since you cleaned up every test Pod before continuing, TOTAL-VIOLATIONS should still be at 0 — the absence of recorded violations confirms no test resource was left uncleaned, not that Gatekeeper stopped auditing.

Summary and next step

This lesson actually installed Kyverno v1.18.2, isolated the test by putting Gatekeeper in dryrun (documenting the exact reason: avoiding attributing a rejection to the wrong engine), applied the equivalent ClusterPolicy in YAML, and confirmed the same result lesson 4 got — Pod with no limits rejected, Pod with limits admitted — with a completely different engine underneath. The comparison table fixed what really matters: for the final result, Gatekeeper and Kyverno are indistinguishable; what changes is only how each team prefers to write the rule.

Before moving on you should be able to: explain why Kyverno's installation uses kubectl create instead of apply; read a Kyverno rejection message and identify its three pieces (webhook, policy/rule, message); and reproduce Step 1 (isolating the test) from memory before installing any new policy engine on a cluster that already has another one active.

Next lesson: hands-on — trivy image over andes-cargo-status-api. There you're going to completely switch layers: instead of evaluating a Kubernetes object's shape before creating it, you're going to scan the content of the Docker image that object references — a completely different artifact from the terraform plan/HCL cloud-security-and-guardrails-guide scanned with the same tool.

Resources

  1. Kyverno — Installation — the source for the official v1.18.2 manifest applied in this lesson, and the recommendation to use kubectl create over kubectl apply.
  2. Kyverno — Background Scans — official documentation for the background field and kyverno-background-controller.
  3. Kyverno — Policy Reports — reference for the PolicyReport generated by kyverno-reports-controller, which you're going to see in lesson 8.
  4. Gatekeeper — Violations — documentation for enforcementAction, contrasted in this lesson with Kyverno's validationFailureAction.
← Previous5. Kyverno: the YAML-native alternativeNext →7. Hands-on: `trivy image` on `andes-cargo-status-api`