Validating Admission Policies in Kyverno

Kyverno, in simple terms, is a policy engine for Kubernetes that can be used to describe policies and validate resource requests against those policies. It allows us to create Policies for our Kubernetes cluster on different levels. It enables us to validate, change, and create resources based on the policies we define.

A Kyverno policy is a collection of rules. Whenever we receive an API request to our Kubernetes cluster, we validate it with a set of rules.
A Policy consists of different clauses such as:

• Match: It selects resources that match the given criteria.
• Exclude: It selects all but excludes resources that match the specified criteria.

Match and Exclude are used to select resources, users, user groups, service accounts, namespaced roles, and cluster-wide roles.

• Validate: It validates the properties of the new resource and it is created if it matches what is declared in the rule.
• Mutate: It modifies matching resources.
• Generate: It creates additional resources.
• Verify Images: It verifies container image signatures using Cosign.

Each rule can contain only a single validate, mutate, generate, or verifyImages child declaration.

In this blog article, I’m going to show how to use validate rules, and how to use CEL expressions for resource validation.

Motivation

Let’s start with creating a Kyverno policy that ensures no hostPath volumes are in use for the newly created pods.

kubectl apply -f - <<EOF
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-host-path
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: host-path
      match:
        any:
        - resources:
            kinds:
              - Deployment
      validate:
        message: >-
                    HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath must be unset.
        pattern:
          spec:
            template:
              spec:
                =(volumes):
                  - X(hostPath): "null"
EOF

spec.rules.pattern is used to define fields and expressions that must be matched by the new resources to pass the validation rule.

pattern:
  spec:
     =(volumes):
       - X(hostPath): "null"

This is read as “if spec.volumesis defined, then spec.volumes.hostPath shouldn’t be defined”. The new pods must match what’s defined in the pattern to be successfully created.

Now, let’s try to deploy an app that uses a hostPath:

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx-server
        image: nginx
        volumeMounts:
          - name: udev
            mountPath: /data
      volumes:
      - name: udev
        hostPath:
          path: /etc/udev
EOF

We can see that our policy is enforced, great!

Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request: 

resource Deployment/default/nginx was blocked due to the following policies 

disallow-host-path:
  host-path: 'validation error: HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath
    must be unset. rule host-path failed at path /spec/template/spec/volumes/0/hostPath/'

An alternative way is to use Common Expression Language (CEL) expressions to validate resources.

CEL was first introduced to Kubernetes for the Validation rules for CustomResourceDefinitions and then it was used by Kubernetes Validating Admission Policies in 1.26.

As a mentee in the LFX mentorship program, I had the opportunity to work on supporting Validating Admission Policies in Kyverno, which is currently in the development phase and scheduled for release in v1.11.

With CEL expressions you can do the following:

• Define policies that take into account multiple fields and values in a resource, and make decisions based on the outcome of those conditions.
• Use logical operators, comparison operators, and custom functions to create policies that are easier to read and understand.
• Define your own custom functions and operators, which can be used to customize the behavior of your policies.

It’s worth considering the strengths and weaknesses of both patterns and CEL expressions and choosing the approach that best suits your needs.

CEL Expressions for Validations in Kyverno

we’ll start with creating a simple policy that uses CEL expressions to validate resources. Then, we’ll create a policy that uses CEL preconditions for fine-grained request filtering. Finally, we’ll create a policy that uses parameter resources.

Create a simple policy to disallow host paths for Deployments

The below policy ensures no hostPath volumes are in use for deployments.

kubectl apply -f - <<EOF
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-host-path
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: host-path
      match:
        any:
        - resources:
            kinds:
              - Deployment
      validate:
        cel:
          expressions:
            - expression: "!has(object.spec.template.spec.volumes) || object.spec.template.spec.volumes.all(volume, !has(volume.hostPath))"
              message: "HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath must be unset."
EOF

spec.rules.validate.cel contains CEL expressions which use the Common Expression Language (CEL) to validate the request. If an expression evaluates to false, the validation check is enforced according to the spec.validationFailureActionfield.

If we tried to deploy an app that uses a hostPath, the resource creation will be blocked as it violates the rule. It’s the same behaviour we saw before.

CEL Preconditions in Kyverno Policies

The below policy ensures the hostPort field is set to a value between 5000 and 6000 for pods whose metadata.name set tonginx

kubectl apply -f - <<EOF
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-host-port-range
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: host-port-range
      match:
        any:
        - resources:
            kinds:
              - Pod
      celPreconditions:
          - name: "first match condition in CEL"
            expression: "object.metadata.name.matches('nginx')"
      validate:
        cel:
          expressions:
          - expression: "object.spec.containers.all(container, !has(container.ports) || container.ports.all(port, !has(port.hostPort) || (port.hostPort >= 5000 && port.hostPort <= 6000)))"
            message: "The only permitted hostPorts are in the range 5000-6000."
EOF

spec.rules.celPreconditions are CEL expressions. All celPreconditions must evaluate to true for the resource to be evaluated. Therefore, any pods that has nginx in its metadata.name will be evaluated.

Now, let’s try to deploy an apache server that contains hostPortset to 80.

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: apache
spec:
  containers:
  - name: apache-server
    image: httpd
    ports:
    - containerPort: 8080
      hostPort: 80
EOF

You’ll see that it’s successfully created because the validation rule wasn’t applied on the new pod as it doesn’t satisfy the celPreconditions. That’s excatly what we need.

pod/apache created

Now, let’s try to deploy an nginx server that contains hostPortset to 80.

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx-server
    image: nginx
    ports:
    - containerPort: 8080
      hostPort: 80
EOF

Since the new pod satisfies the celPreconditions, therefore the validation rule will be applied on it. As a result, the creation of the pod will be blocked as it violates the rule.

Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request: 

resource Pod/default/nginx was blocked due to the following policies 

disallow-host-port-range:
  host-port-range: The only permitted hostPorts are in the range 5000-6000.

Parameter Resources in Kyverno Policies

The below policy ensures the deployment replicas is less than a specific value. This value is defined in a paramter resource.

kubectl apply -f - <<EOF
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: check-deployment-replicas
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: deployment-replicas
      match:
        any:
        - resources:
            kinds:
              - Deployment
      validate:
        cel:
          paramKind: 
            apiVersion: rules.example.com/v1
            kind: ReplicaLimit
          paramRef:
            name: "replica-limit-test.example.com"
          expressions:
            - expression: "object.spec.replicas <= params.maxReplicas"
              messageExpression:  "'Deployment spec.replicas must be less than ' + string(params.maxReplicas)"
EOF

The cel.paramKind and cel.paramRefspecifies the kind of resources used to parameterize this policy. For this example, it is configured by ReplicaLimitcustom resources.

The ReplicaLimitcould be as follows:

kubectl apply -f - <<EOF
apiVersion: rules.example.com/v1
kind: ReplicaLimit
metadata:
  name: "replica-limit-test.example.com"
maxReplicas: 3
EOF

Now, let’s try to deploy an app with 5 replicas.

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 5
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx-server
        image: nginx
EOF

As we expect, the deployment creation will be blocked because it violates the rule.

Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request: 

resource Deployment/default/nginx was blocked due to the following policies 

check-deployment-replicas:
  deployment-replicas: Deployment spec.replicas must be less than 3

Now, let’s try to deploy an app with 2 replicas.

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx-server
        image: nginx
EOF

The deployment is created successfully, Great!

deployment.apps/nginx created

Validating Admission Policies using Kyverno CLI

The Kyverno Command Line Interface (CLI) is designed to validate and test policy behavior to resources prior to adding them to a cluster.

One of the Kyverno CLI commands are apply and test commands.

• The apply command is used to perform a dry run on one or more policies for the given manifest(s).
• The test command is used to test a given set of resources against one or more policies to check the desired results.

Kyverno CLI now can be used to apply/test Kubernetes Validating Admission Policies that were introduced in 1.26

Validating Admission Policies using Kyverno apply

Here’s a Validating Admission Policy:

cat > check-deployment-replicas.yaml
apiVersion: admissionregistration.k8s.io/v1alpha1
kind: ValidatingAdmissionPolicy
metadata:
  name: check-deployments-replicas
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups:   ["apps"]
      apiVersions: ["v1"]
      operations:  ["CREATE", "UPDATE"]
      resources:   ["deployments"]
  validations:
    - expression: "object.spec.replicas <= 2"
      message: "Replicas must be less than or equal 2"

A deployment can be:

cat > deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx-server
        image: nginx

Let’s apply the policy on the resource using kyverno apply as follows:

kyverno apply ./check-deployment-replicas.yaml --resource=deployment.yaml

pass: 1, fail: 0, warn: 0, error: 0, skip: 0 

In case we changed the replicas to 3, you’ll see that there’s a failed policy.

kyverno apply ./check-deployment-replicas.yaml --resource=deployment.yaml

pass: 0, fail: 1, warn: 0, error: 0, skip: 0

Validating Admission Policies using Kyverno test

Lets’ try to test two deployments aganist the above validating admission policy and check the desired results.

cat > deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-1
  labels:
    app: nginx-1
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-1
  template:
    metadata:
      labels:
        app: nginx-1
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-2
  labels:
    app: nginx-2
spec:
  replicas: 4
  selector:
    matchLabels:
      app: nginx-2
  template:
    metadata:
      labels:
        app: nginx-2
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

The tests are written in a file name kyverno-test.yaml so we will create two tests, one for each deployment and test them aganist the policy.

cat > kyverno-test.yaml
name: check-deployment-replicas-test
policies:
  - check-deployment-replicas.yaml
resources:
  - deployment.yaml
results:
  - policy: check-deployment-replicas
    isVap: true
    resource: nginx-deployment-1
    kind: Deployment
    result: pass
  - policy: check-deployment-replicas
    isVap: true
    resource: nginx-deployment-2
    kind: Deployment
    result: fail

Now, we’re ready to test the two deployments aganist validating admission policy. A directory test-dir is created and it contains the above manifests:

1. check-deployment-replicas.yaml
2. deployment.yaml
3. kyverno-test.yaml

kyverno test ./test-dir/

Executing check-deployment-replicas-test...

│────│───────────────────────────│──────│────────────────────────────────│────────│
│ ID │ POLICY                    │ RULE │ RESOURCE                       │ RESULT │
│────│───────────────────────────│──────│────────────────────────────────│────────│
│  1 │ check-deployment-replicas │      │ /Deployment/nginx-deployment-1 │ Pass   │
│  2 │ check-deployment-replicas │      │ /Deployment/nginx-deployment-2 │ Pass   │
│────│───────────────────────────│──────│────────────────────────────────│────────│

Test Summary: 2 tests passed and 0 tests failed

As expected, the two tests passed because the actual result of each test matches the desired result as defined in the test manifest.

Conclusion

This blog post highlights the use of CEL expressions in Kyverno validation rules as well as the use of Kyverno CLI apply/test commands for Kubernetes Validating Admission policies.

Visit kyverno.io to explore more!