Using Kyverno policies with ArgoCD

Kyverno and ArgoCD are two great Kubernetes tools.

Kyverno is a Kubernetes policy engine that can be used to enforce security Kyverno.

ArgoCD is a continuous delivery solution implementing the GitOps approach.

By combining ArgoCD and Kyverno, we can declare policies using standard Kubernetes manifests in a git repository and get them applied to Kubernetes clusters automatically.

When a policy changes in the git repository, ArgoCD detects the change and reconciles the desired state with actual state making the cluster converge to the state described in git.

This sounds pretty straightforward but Kyverno comes with a mutating webhook that will generate additional rules in a policy before it is applied and this will confuse ArgoCD.

ArgoCD will constantly see a difference between the desired and actual states because of the rules that have been added on the fly.

Let’s see this in practice with the following policy:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-seccomp
spec:
  background: true
  validationFailureAction: audit
  rules:
    - match:
        any:
          - resources:
              kinds:
                - Pod
      name: check-seccomp
      validate:
        pattern:
          spec:
            =(ephemeralContainers):
              - =(securityContext):
                  =(seccompProfile):
                    =(type): RuntimeDefault | Localhost
            =(initContainers):
              - =(securityContext):
                  =(seccompProfile):
                    =(type): RuntimeDefault | Localhost
            =(securityContext):
              =(seccompProfile):
                =(type): RuntimeDefault | Localhost
            containers:
              - =(securityContext):
                  =(seccompProfile):
                    =(type): RuntimeDefault | Localhost

When the policy above is applied, the Kyverno webhook will add generated rules, resulting in the following policy:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-seccomp
spec:
  background: true
  validationFailureAction: audit
  failurePolicy: Fail
  rules:
    - match:
        any:
          - resources:
              kinds:
                - Pod
      name: check-seccomp
      validate:
        pattern:
          spec:
            =(ephemeralContainers):
              - =(securityContext):
                  =(seccompProfile):
                    =(type): RuntimeDefault | Localhost
            =(initContainers):
              - =(securityContext):
                  =(seccompProfile):
                    =(type): RuntimeDefault | Localhost
            =(securityContext):
              =(seccompProfile):
                =(type): RuntimeDefault | Localhost
            containers:
              - =(securityContext):
                  =(seccompProfile):
                    =(type): RuntimeDefault | Localhost
    - match:
        any:
          - resources:
              kinds:
                - DaemonSet
                - Deployment
                - Job
                - StatefulSet
        resources: {}
      name: autogen-check-seccomp
      validate:
        pattern:
          spec:
            template:
              spec:
                =(ephemeralContainers):
                  - =(securityContext):
                      =(seccompProfile):
                        =(type): RuntimeDefault | Localhost
                =(initContainers):
                  - =(securityContext):
                      =(seccompProfile):
                        =(type): RuntimeDefault | Localhost
                =(securityContext):
                  =(seccompProfile):
                    =(type): RuntimeDefault | Localhost
                containers:
                  - =(securityContext):
                      =(seccompProfile):
                        =(type): RuntimeDefault | Localhost
    - match:
        any:
          - resources:
              kinds:
                - CronJob
        resources: {}
      name: autogen-cronjob-check-seccomp
      validate:
        pattern:
          spec:
            jobTemplate:
              spec:
                template:
                  spec:
                    =(ephemeralContainers):
                      - =(securityContext):
                          =(seccompProfile):
                            =(type): RuntimeDefault | Localhost
                    =(initContainers):
                      - =(securityContext):
                          =(seccompProfile):
                            =(type): RuntimeDefault | Localhost
                    =(securityContext):
                      =(seccompProfile):
                        =(type): RuntimeDefault | Localhost
                    containers:
                      - =(securityContext):
                          =(seccompProfile):
                            =(type): RuntimeDefault | Localhost

Without surprise, ArgoCD will report that the policy is OutOfSync.

In order to make ArgoCD happy, we need to ignore the generated rules. Fortunately we can do just that using the ignoreDifferences stanza of an Application spec.

Diffing Customization - Argo CD - Declarative GitOps CD for Kubernetes

The application below deploys the kyverno-policies helm chart without specifying ignoreDifferences and therefore will suffer the continuous OutOfSync symptoms:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kyverno-policies
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://kyverno.github.io/kyverno
    chart: kyverno-policies
    targetRevision: '*'
  destination:
    server: https://kubernetes.default.svc
    namespace: kyverno

To fix the issue, we need to fill in the ignoreDifferences stanza in the Application spec with the correct path expression to match only generated rules.

We will use a JQ path expression to select the generated rules we want to ignore:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kyverno-policies
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://kyverno.github.io/kyverno
    chart: kyverno-policies
    targetRevision: '*'
  destination:
    server: https://kubernetes.default.svc
    namespace: kyverno
  ignoreDifferences:
    - group: kyverno.io
      kind: ClusterPolicy
      jqPathExpressions:
        - .spec.rules[] | select(.name|test("autogen-."))

Now, all generated rules will be ignored by ArgoCD, and Kyverno policies will be correctly kept in sync in the target cluster 🎉

Please note that you can also configure ignore differences at the system level to make ArgoCD ignore ClusterPolicy and Policy generated rules globally without specifying ignoreDifferences stanza in Application spec.

Refer to ArgoCD documentation for configuring ignore differences at the system level.