Using Kyverno policies with ArgoCD
Kyverno and ArgoCD are two great Kubernetes tools.
Kyverno is a Kubernetes policy engine that can be used to enforce security Kyverno.
ArgoCD is a continuous delivery solution implementing the GitOps approach.
By combining ArgoCD and Kyverno, we can declare policies using standard Kubernetes manifests in a git repository and get them applied to Kubernetes clusters automatically.
When a policy changes in the git repository, ArgoCD detects the change and reconciles the desired state with actual state making the cluster converge to the state described in git.
This sounds pretty straightforward but Kyverno comes with a mutating webhook that will generate additional rules in a policy before it is applied and this will confuse ArgoCD.
ArgoCD will constantly see a difference between the desired and actual states because of the rules that have been added on the fly.
Let’s see this in practice with the following policy:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: restrict-seccomp
spec:
background: true
validationFailureAction: audit
rules:
- match:
any:
- resources:
kinds:
- Pod
name: check-seccomp
validate:
pattern:
spec:
=(ephemeralContainers):
- =(securityContext):
=(seccompProfile):
=(type): RuntimeDefault | Localhost
=(initContainers):
- =(securityContext):
=(seccompProfile):
=(type): RuntimeDefault | Localhost
=(securityContext):
=(seccompProfile):
=(type): RuntimeDefault | Localhost
containers:
- =(securityContext):
=(seccompProfile):
=(type): RuntimeDefault | Localhost
When the policy above is applied, the Kyverno webhook will add generated rules, resulting in the following policy:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: restrict-seccomp
spec:
background: true
validationFailureAction: audit
failurePolicy: Fail
rules:
- match:
any:
- resources:
kinds:
- Pod
name: check-seccomp
validate:
pattern:
spec:
=(ephemeralContainers):
- =(securityContext):
=(seccompProfile):
=(type): RuntimeDefault | Localhost
=(initContainers):
- =(securityContext):
=(seccompProfile):
=(type): RuntimeDefault | Localhost
=(securityContext):
=(seccompProfile):
=(type): RuntimeDefault | Localhost
containers:
- =(securityContext):
=(seccompProfile):
=(type): RuntimeDefault | Localhost
- match:
any:
- resources:
kinds:
- DaemonSet
- Deployment
- Job
- StatefulSet
resources: {}
name: autogen-check-seccomp
validate:
pattern:
spec:
template:
spec:
=(ephemeralContainers):
- =(securityContext):
=(seccompProfile):
=(type): RuntimeDefault | Localhost
=(initContainers):
- =(securityContext):
=(seccompProfile):
=(type): RuntimeDefault | Localhost
=(securityContext):
=(seccompProfile):
=(type): RuntimeDefault | Localhost
containers:
- =(securityContext):
=(seccompProfile):
=(type): RuntimeDefault | Localhost
- match:
any:
- resources:
kinds:
- CronJob
resources: {}
name: autogen-cronjob-check-seccomp
validate:
pattern:
spec:
jobTemplate:
spec:
template:
spec:
=(ephemeralContainers):
- =(securityContext):
=(seccompProfile):
=(type): RuntimeDefault | Localhost
=(initContainers):
- =(securityContext):
=(seccompProfile):
=(type): RuntimeDefault | Localhost
=(securityContext):
=(seccompProfile):
=(type): RuntimeDefault | Localhost
containers:
- =(securityContext):
=(seccompProfile):
=(type): RuntimeDefault | Localhost
Without surprise, ArgoCD will report that the policy is OutOfSync
.
In order to make ArgoCD happy, we need to ignore the generated rules. Fortunately we can do just that using the ignoreDifferences
stanza of an Application
spec.
The application below deploys the kyverno-policies
helm chart without specifying ignoreDifferences
and therefore will suffer the continuous OutOfSync
symptoms:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: kyverno-policies
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: https://kyverno.github.io/kyverno
chart: kyverno-policies
targetRevision: '*'
destination:
server: https://kubernetes.default.svc
namespace: kyverno
To fix the issue, we need to fill in the ignoreDifferences
stanza in the Application
spec with the correct path expression to match only generated rules.
We will use a JQ path expression to select the generated rules we want to ignore:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: kyverno-policies
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: https://kyverno.github.io/kyverno
chart: kyverno-policies
targetRevision: '*'
destination:
server: https://kubernetes.default.svc
namespace: kyverno
ignoreDifferences:
- group: kyverno.io
kind: ClusterPolicy
jqPathExpressions:
- .spec.rules[] | select(.name|test("autogen-."))
Now, all generated rules will be ignored by ArgoCD, and Kyverno policies will be correctly kept in sync in the target cluster 🎉
Please note that you can also configure ignore differences at the system level to make ArgoCD ignore ClusterPolicy
and Policy
generated rules globally without specifying ignoreDifferences
stanza in Application
spec.
Refer to ArgoCD documentation for configuring ignore differences at the system level.