5-Step Approach: Projectsveltos for Kubernetes add-on deployment and management on RKE2

Working with many different Kubernetes add-on deployments, the actual deployment and management of those across different clusters, on-prem and in the Cloud, can be challenging and sometimes frustrating.

Projectsveltos is a Kubernetes add-on controller that simplifies the deployment and management of different add-ons and applications across multiple clusters (on-prem, Cloud). Sveltos runs in a management cluster and programmatically deploys and manages add-ons and applications on any cluster in the fleet, including the management cluster itself. Sveltos supports many add-on formats, including Helm charts, raw YAML/JSON, Kustomize, Carvel ytt, and Jsonnet.

In this blog post, we will demonstrate how easy and fast it is to deploy Sveltos on an RKE2 cluster with the help of ArgoCD, register two RKE2 Cluster API (CAPI) clusters and create a ClusterProfile to deploy Prometheus and Grafana Helm charts down the managed CAPI clusters.

Diagram

Projectsveltos Demo Diagram

Prerequisites

For this demonstration, I have already installed ArgoCD on a central cluster. If you would like to learn more about the ArgoCD installation, go through the official documentation found here. If you would like to follow along, below you can find the lab details used.

+-----------------+----------------------+------------------------+
|  Cluster Name   |         Type         |        Version         |
+-----------------+----------------------+------------------------+
|    cluster04    | Management Cluster   | RKE2 v1.26.11+rke2r1   |
|    cluster12    | Managed CAPI Cluster | RKE2 v1.26.6+rke2r1    |
|    cluster13    | Managed CAPI Cluster | RKE2 v1.26.6+rke2r1    |
+-----------------+----------------------+------------------------+

Step 1: Deploy Sveltos as a Helm Chart cluster04

Sveltos can be deployed either as a manifest or as a Helm chart. For more information about the different installation options, check the link here. In my case, I chose to follow the GitOps approach and let ArgoCD deal with the comparison and synchronisation of the Git repository where the code to deploy Sveltos is stored, with the actual running application.

If you are unsure how to deploy Helm charts with ArgoCD, have a look here.

Verification

After we deploy Sveltos, we want to ensure everything is in a working and fully functional state. This can be achieved either from the ArgoCD UI or from the management cluster itself.

ArgoCD — Sveltos Helm Chart Deployment

$ kubectl get pods -n projectsveltos

NAME                                        READY   STATUS    RESTARTS   AGE
access-manager-77c7c64477-ns8ml             2/2     Running   0          70s
addon-compliance-manager-7f449d884c-6kgqr   2/2     Running   0          69s
addon-controller-55d7d848ff-ps8l8           2/2     Running   0          70s
classifier-manager-67d6f67d5b-cgpr7         2/2     Running   0          70s
event-manager-69db45b65d-htz5l              2/2     Running   0          70s
hc-manager-5679c69dcc-z6s48                 2/2     Running   0          70s
sc-manager-84dbd64fb4-6hwpf                 2/2     Running   0          70s
shard-controller-56678bcf8c-zjbvc           2/2     Running   0          70s

Step 2: Install the Sveltosctl

The Sveltosctl, is the command-line interface (CLI) for Sveltos. This is an available option to query Sveltos resources and it is available as a Kubernetes pod or as a binary.

As I would like to register cluster12 and cluster13 to the Sveltos management cluster, the Sveltosctl as a binary will be used.

Step 3: Register CAPI Clusters with Sveltos

To register any cluster with Sveltos, you only need three things:

1. A ServiceAccount for Sveltos and a kubeconfig associated with that account;
2. A namespace where you want to register the external cluster;
3. The Sveltosctl should point to the management cluster and then perform the ‘sveltosctl register cluster’ command.

Now, if you are unsure how to create a Service Account and an associated kubeconfig, do not worry. There is a script publicly available to create everything you need automatically.

Registration

$ sveltosctl register cluster --namespace=projectsveltos --cluster=cluster12 --kubeconfig=cluster12.yaml

$ sveltosctl register cluster --namespace=projectsveltos --cluster=cluster13 --kubeconfig=cluster13.yaml

From the commands above, we register cluster12 and cluster13 in the namespace projectsveltos. Of course, you can register the clusters to a namespace of your preference.

Verification

$ kubectl get sveltosclusters -n projectsveltos

NAME        READY   VERSION
cluster12   true    v1.26.6+rke2r1
cluster13   true    v1.26.6+rke2r1

Step 4: Cluster Labelling

To allow Sveltos to deploy and manage Kubernetes add-ons, the concept of ClusterProfile and cluster labelling comes into play. ClusterProfile is the CustomerResourceDefinition used to instruct Sveltos which add-ons to deploy on a set of clusters.

For this demonstration, we will set the label “env:prod” to both Sveltos clusters. The below commands are executed on the management cluster (cluster04).

$ kubectl get sveltosclusters -n projectsveltos --show-labels

NAME        READY   VERSION          LABELS
cluster12   true    v1.26.6+rke2r1   sveltos-agent=present
cluster13   true    v1.26.6+rke2r1   sveltos-agent=present

$ kubectl label sveltosclusters cluster12 env=prod -n projectsveltos

$ kubectl label sveltosclusters cluster13 env=prod -n projectsveltos

$ kubectl get sveltosclusters -n projectsveltos --show-labels

NAME        READY   VERSION          LABELS
cluster12   true    v1.26.6+rke2r1   env=prod,sveltos-agent=present
cluster13   true    v1.26.6+rke2r1   env=prod,sveltos-agent=present

Step 5: ClusterProfile for Grafana and Prometheus

The below ClusterProfile is an example of a Helm chart deployment of Grafana and Prometheus to Sveltos clusters with the label set to “env:prod”.

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: prometheus-grafana
spec:
  clusterSelector: env=prod
  helmCharts:
  - repositoryURL:    https://prometheus-community.github.io/helm-charts
    repositoryName:   prometheus-community
    chartName:        prometheus-community/prometheus
    chartVersion:     23.4.0
    releaseName:      prometheus
    releaseNamespace: prometheus
    helmChartAction:  Install
  - repositoryURL:    https://grafana.github.io/helm-charts
    repositoryName:   grafana
    chartName:        grafana/grafana
    chartVersion:     6.58.9
    releaseName:      grafana
    releaseNamespace: grafana
    helmChartAction:  Install

Apply the ClusterProfile

$ kubectl apply -f "grafana_prometheus.yaml"

Once the ClusterProfile is applied to the management cluster, the expected result is to have the Grafana and the Prometheus deployment on both managed clusters.

Verification

$ sveltosctl show addons

+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+
|         CLUSTER          | RESOURCE TYPE | NAMESPACE  |    NAME    | VERSION |             TIME              |  CLUSTER PROFILES  |
+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+
| projectsveltos/cluster12 | helm chart    | prometheus | prometheus | 23.4.0  | 2023-12-17 11:25:20 +0100 CET | prometheus-grafana |
| projectsveltos/cluster12 | helm chart    | grafana    | grafana    | 6.58.9  | 2023-12-17 11:25:23 +0100 CET | prometheus-grafana |
| projectsveltos/cluster13 | helm chart    | prometheus | prometheus | 23.4.0  | 2023-12-17 11:25:30 +0100 CET | prometheus-grafana |
| projectsveltos/cluster13 | helm chart    | grafana    | grafana    | 6.58.9  | 2023-12-17 11:25:32 +0100 CET | prometheus-grafana |
+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+

Verification — Cluster12

$ kubectl get pods -n grafana

NAME                           READY   STATUS    RESTARTS   AGE
pod/grafana-78764f9cd6-zsqdx   1/1     Running   0          81s

$ kubectl pods all -n prometheus

NAME                                                     READY   STATUS    RESTARTS   AGE
pod/prometheus-alertmanager-0                            1/1     Running   0          2m3s
pod/prometheus-kube-state-metrics-587bd996f6-l94zq       1/1     Running   0          2m3s
pod/prometheus-prometheus-node-exporter-khw75            1/1     Running   0          2m3s
pod/prometheus-prometheus-pushgateway-75986b9c9f-2ql7v   1/1     Running   0          2m3s
pod/prometheus-server-86c66b89c6-7xk9r                   2/2     Running   0          2m3s

The same verification can be performed for cluster13.

Remove Label ‘env:prod’ cluster12

You might wonder what will happen if we remove the label ‘env:prod’ from either cluster12 or cluster13. The answer is that Sveltos will identify the missing label ‘env:prod’ and undeploy the Grafana and the Prometheus deployment from the cluster.

Let’s have a look.

Remove Label

$ kubectl label sveltosclusters cluster12 env- -n projectsveltos

Verification

$ kubectl get sveltosclusters -n projectsveltos --show-labels

NAME        READY   VERSION          LABELS
cluster12   true    v1.26.6+rke2r1   sveltos-agent=present
cluster13   true    v1.26.6+rke2r1   env=prod,sveltos-agent=present

sveltosctl show addons

+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+
|         CLUSTER          | RESOURCE TYPE | NAMESPACE  |    NAME    | VERSION |             TIME              |  CLUSTER PROFILES  |
+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+
| projectsveltos/cluster13 | helm chart    | grafana    | grafana    | 6.58.9  | 2023-12-17 11:25:32 +0100 CET | prometheus-grafana |
| projectsveltos/cluster13 | helm chart    | prometheus | prometheus | 23.4.0  | 2023-12-17 11:25:30 +0100 CET | prometheus-grafana |
+--------------------------+---------------+------------+------------+---------+-------------------------------+--------------------+

As expected, Sveltos removed the deployments. The same will happen if we register a new cluster and assign the label ‘env:prod’. Sveltos will take care of the complete lifecycle of your Kubernetes deployments in a simple and straightforward manner.

Contact

We are here to help! Whether you have questions, issues or need assistance, our Slack channel is the perfect place for you. Click here to join us.

👏 Support this project

Every contribution counts! If you enjoyed this article, check out the Projectsveltos GitHub repo. You can star 🌟 the project if you found it helpful.

The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and many more examples.

Thanks for reading!