5-Step Approach: ProjectSveltos Integration with Flux
Introduction
To allow platform administrators more flexibility with different Kubernetes deployments, the latest Sveltos release v0.23.0 enhanced the integration with Flux. Flux is a CNCF graduate project that offers users a set of continuous and progressive delivery solutions for Kubernetes which are open and extensible.
With the integration in place, we can automate the synchronisation of desired Kubernetes add-ons, removing any manual steps and ensuring consistent deployments across clusters. Platform administrators are not forced to store Kubernetes manifests in the management cluster but rather in a Git repository or a Bucket.
In today’s post, we will demonstrate how easy it is to use Flux and Sveltos to deploy different Kyverno policies to an RKE2 managed cluster with the label selector set to end=test.
Diagram
Prerequisites
To follow along, it is strongly advised to take a look at the previous post detailing the deployment of Sveltos on a cluster. To install Sveltos, just follow the instructions outlined in steps 1 and 2 of the mentioned post.
Lab Setup
+-----------------+-------------------+--------------------------+
| Cluster Name | Type | Version |
+-----------------+-------------------+--------------------------+
| cluster09 | Management Cluster| RKE2 v1.26.12+rke2r1 |
| rke2-demo01 | Managed Cluster | RKE2 v1.26.14+rke2r1 |
+-----------------+-------------------+--------------------------+Step 1: Register and label the RKE2 cluster with Sveltos
Once the RKE2 cluster is in a “Running” state, we will use the sveltosctl to register it. For the registration, we need three things: a service account, a kubeconfig associated with that account and a namespace. If you are unsure how to create a Service Account and an associated kubeconfig, there is a script publicly available to help you out.
Registration
$ sveltosctl register cluster --namespace=projectsveltos --cluster=rke2-demo01 --kubeconfig=rke2-demo01.yaml$ kubectl get sveltoscluster -n projectsveltos
NAME READY VERSION
projectsveltos rke2-demo01 true v1.26.14+rke2r1Assing Label
$ kubectl label sveltoscluster rke2-demo01 env=test -n projectsveltos$ kubectl get sveltoscluster -n projectsveltos --show-labels
NAME READY VERSION LABELS
projectsveltos rke2-demo01 true v1.26.14+rke2r1 env=test,sveltos-agent=presentStep 2: Download and Install Flux
For this demonstration, we will download the Flux CLI binary for an Ubuntu 22.04 VM, extract it and create a Kubernetes secret to allow synchronisation with a respective GitHub repository. For more information about the Flux installation, have a look here.
Download and Install Flux CLI
$ curl -L -o flux_2.2.3_linux_amd64.tar.gz https://github.com/fluxcd/flux2/releases/download/v2.2.3/flux_2.2.3_linux_amd64.tar.gz # Download the respective dinary
$ tar -xvf flux_2.2.3_linux_amd64.tar.gz # Extract the binary
$ sudo mv flux /usr/bin # Move the binary to the respective common binary Linux directoryStep 3: Flux Bootstrap (management cluster)
Create GitHub Token
As we will instruct Flux to synchronise with a GitHub repository, it should have all the right read/write permissions. Have a look here for more information about the bootstrap process and here for the access token creation.
Export GitHub Token
$ export GITHUB_TOKEN=ghp_YOUR_OWN_TOKENCreate Flux GitRepository Resource
./flux bootstrap github \
--token-auth \
--owner={your GitHub username} \
--repository={your Repo name} \
--branch=main \
--path={where Flux should store related files} \
--personalExample GitRepository Resource
./flux bootstrap github \
--token-auth \
--owner=myusername \
--repository=yaml_flux \
--branch=main \
--path=clusters/test-cluster \
--personalVerification
$ kubectl get gitrepository.source.toolkit.fluxcd.io -A
NAMESPACE NAME URL AGE READY STATUS
flux-system flux-system https://github.com/egrosdou01/yaml_flux.git 9m47s True stored artifact for revision 'main@sha1:65bdbe4f1a41f6b5c426b7cd4d0517b7f2941361'(Optional) Step 4: Update GitRepository Resource
To allow Sveltos to work with Flux and apply a YAML template, we have to update the GitRepository resource created in a previous step to include the projectsveltos.io/template: "true" annotation.
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
annotations:
projectsveltos.io/template: "true"
creationTimestamp: "2024-03-22T11:56:44Z"
finalizers:
- finalizers.fluxcd.io
generation: 1
labels:
kustomize.toolkit.fluxcd.io/name: flux-system
kustomize.toolkit.fluxcd.io/namespace: flux-system
name: flux-system
namespace: flux-system
resourceVersion: "24357"
uid: 4dd5400e-67e8-4622-a696-e92fd6d22c51
spec:
interval: 1m0s
ref:
branch: main
secretRef:
name: flux-system
timeout: 60s
"/tmp/kubectl-edit-2451969467.yaml" 50L, 1786C Note: If you see errors related to the unreachability of the Git repository specified, have a look at the Flux network policies applied by default and during the installation. Use the commands below for verification.
$ kubectl get netpol -n flux-system
NAME POD-SELECTOR AGE
allow-egress <none> 2d23h
allow-scraping <none> 2d23h
allow-webhooks app=notification-controller 2d23hStep 5: Deploy Kyverno and Kyverno Policies (managed cluster)
It is time to deploy Kyverno and two related policies to the managed cluster. For that purpose, we will create a Sveltos ClusterProfile. The ClusterProfile will install Kyverno as a Helm chart and once this is successful, it will continue with a top-down approach to deploy the related Kyverno policies.
ClusterProfile
---
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
name: kyverno
spec:
clusterSelector: env=test
syncMode: Continuous
helmCharts:
- repositoryURL: https://kyverno.github.io/kyverno/
repositoryName: kyverno
chartName: kyverno/kyverno
chartVersion: v3.1.4
releaseName: kyverno-latest
releaseNamespace: kyverno
helmChartAction: Install
validateHealths:
- name: deployment-health
featureID: Helm
group: "apps"
version: "v1"
kind: "Deployment"
namespace: kyverno
script: |
function evaluate()
local hs = {healthy = false, message = "Available replicas not match requested replicas"}
if obj.status and obj.status.availableReplicas ~= nil and obj.status.availableReplicas == obj.spec.replicas then
hs.healthy = true
end
return hs
end
policyRefs:
- kind: GitRepository
name: flux-system
namespace: flux-system
path: kyverno-policies
deploymentType: RemoteThe above ClusterProfile will deploy the Kyverno Helm chart version 3.1.4 to the managed cluster, and once the deployment is declared as “Healthy” based on the Lua language code, we will proceed with the Kyverno policies located inside the kyverno-policies/ directory within the GitHub repository. Example Kyverno policies can be found here.
Apply ClusterProfile
$ kubectl apply -f clusterprofile-kyverno.yamlVerify (management cluster)
Sveltos will create the clusterprofile and the clustersummary Kubernetes resources after the deployment of the ClusterProfile.
$ kubectl get clusterprofile,clustersummary -n projectsveltos
NAME AGE
clusterprofile.config.projectsveltos.io/kyverno 56s
NAMESPACE NAME AGE
projectsveltos clustersummary.config.projectsveltos.io/kyverno-sveltos-rke2-demo01 56s$ sveltosctl show addons
+----------------------------+--------------------------+-----------+---------------------+---------+-------------------------------+------------------+
| CLUSTER | RESOURCE TYPE | NAMESPACE | NAME | VERSION | TIME | CLUSTER PROFILES |
+----------------------------+--------------------------+-----------+---------------------+---------+-------------------------------+------------------+
| projectsveltos/rke2-demo01 | helm chart | kyverno | kyverno-latest | 3.1.4 | 2024-03-25 15:29:02 +0000 UTC | kyverno |
| projectsveltos/rke2-demo01 | kyverno.io:ClusterPolicy | | disallow-latest-tag | N/A | 2024-03-25 15:29:52 +0000 UTC | kyverno |
| projectsveltos/rke2-demo01 | kyverno.io:ClusterPolicy | | require-ro-rootfs | N/A | 2024-03-25 15:29:52 +0000 UTC | kyverno |
+----------------------------+--------------------------+-----------+---------------------+---------+-------------------------------+------------------+From the above output, we do see that Sveltos first deployed the Kyverno Helm chart and after a minute or so the respective policies.
Verify (managed cluster)
$ kubectl get kyverno -A
NAME ADMISSION BACKGROUND VALIDATE ACTION READY AGE MESSAGE
clusterpolicy.kyverno.io/disallow-latest-tag true true audit True 15m Ready
clusterpolicy.kyverno.io/require-ro-rootfs true true audit True 15m ReadyFor the verification on the managed cluster, we can create a new pod that uses the busybox:latest image.
$ kubectl run busybox --image=busybox:latest -- sleep 3600Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 16s default-scheduler Successfully assigned default/busybox to eleni-cluster-test-pool2-4522577b-ls7fh
Warning PolicyViolation 16s (x2 over 16s) kyverno-admission policy disallow-latest-tag/validate-image-tag fail: validation error: Using a mutable image tag e.g. 'latest' is not allowed. rule validate-image-tag failed at path /spec/containers/0/image/
Normal Pulled 13s kubelet Successfully pulled image "busybox:latest" in 2.603318517s (2.603331706s including waiting)
Normal Pulling 12s (x2 over 16s) kubelet Pulling image "busybox:latest"
Normal Created 11s (x2 over 13s) kubelet Created container busybox
Normal Started 11s (x2 over 13s) kubelet Started container busybox
Normal Pulled 11s kubelet Successfully pulled image "busybox:latest" in 879.087453ms (879.104025ms including waiting)
Warning BackOff 9s (x2 over 10s) kubelet Back-off restarting failed container busybox in pod busybox_default(06688eec-d3e8-4e78-ab79-2dec5519b8f2) A similar test can be performed for a container with root access.
Resources
- Sveltos: https://projectsveltos.github.io/sveltos/
- Sveltos Flux Integration: https://projectsveltos.github.io/sveltos/use_cases/use_case_flux_integration/
- Flux: https://fluxcd.io/
Contact
We are here to help! Whether you have questions, issues or need assistance, our Slack channel is the perfect place for you. Click here to join us.
👏 Support this project
Every contribution counts! If you enjoyed this article, check out the Projectsveltos GitHub repo. You can star 🌟 the project if you find it helpful.
The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and many more examples.
Thanks for reading!
