ArgoCD: Multi-cluster Helm charts management in mono-repo

Introduction

When using ArgoCD to manage multiple clusters, you may have issues to support multiple versions of the same component accross different environments. Let’s take some example:

• Experiment operator on dev environment: You have 3 environments dev, uat and prod. You may want to experiment an operator or release candidate in dev environment without impacting other environments
• Rolling upgrade: Your operator is installed in the 3 environments. A new release is out and you want to smoothly roll it out from dev to production
• Work in parallel: Multiple developer may want to work on different operators

This article will expose you a method to manage your dependencies in a mono-repo with only one revision (HEAD)

You need to be familiar with the ArgoCD:

• Multiple sources for an Application feature.
• ApplicationSet feature

Requirements

Let’s first expose the requirements

• We want a mono repo with only one revision for all clusters (HEAD)
• We want all value files collocated
• We want auto-discovery of new chart
• We want to be able to enable/disable a chart per environment
• Rolling upgrade: We want to be able to manage different version of the chart / values per environment

Proposed solution

The solution requires a folder (let’s say public-charts) that contains one folder per chart. Each chart folder contains one value file per environment and an .argocd.json. Here is an example with some charts that are widely used:

.argocd.json

In the above screenshot you may have noticed an .argocd.json file.

At Ubisoft Data Platform Group we decided to introduce and generalise an .argocd.json file.
This file is a descriptor that defines:

• Chart & revision to use accross all environment
• Destination namespace
• Sync Options (ie: serverSideApply)
• Environment breakdown (activation, revision override, options override)

It will be discovered by an ApplicationSet to generate one Application per chart to install.

Here is a simple example:

{
    "source": {
        "repoURL": "https://kyverno.github.io/kyverno",
        "targetRevision": "2.7.0"
    },
    "destination": {
        "namespace": "kyverno"
    },
    "clusters": {
        "dev": {
            "enabled": true
        },
        "uat": {
            "enabled": true
        },
        "prod": {
            "enabled": true
        }
    }
}

This instructs to deploy kyverno chart 2.7.0 into namespace kyverno. Until here no complexity.

Uninstall a component from environment

{
    "source": {
        "repoURL": "https://kyverno.github.io/kyverno",
        "targetRevision": "2.7.0"
    },
    "destination": {
        "namespace": "kyverno"
    },
    "clusters": {
        "dev": {
            "enabled": false // Operator will be uninstalled on dev environment
        },
        "uat": {
            "enabled": true
        },
        "prod": {
            "enabled": true
        }
    }
}

Upgrade a component version in an environment

{
    "source": {
        "repoURL": "https://kyverno.github.io/kyverno",
        "targetRevision": "2.7.0"
    },
    "destination": {
        "namespace": "kyverno"
    },
    "clusters": {
        "dev": {
            "enabled": true,
            "chartRevision": "2.8.0" // this override the chart revision for an upgrade in dev only
        },
        "uat": {
            "enabled": true
        },
        "prod": {
            "enabled": true
        }
    }
}

You can then rollout your environments

{
    "source": {
        "repoURL": "https://kyverno.github.io/kyverno",
        "targetRevision": "2.7.0"
    },
    "destination": {
        "namespace": "kyverno"
    },
    "clusters": {
        "dev": {
            "enabled": true,
            "chartRevision": "2.8.0"
        },
        "uat": {
            "enabled": true,
            "chartRevision": "2.8.0"
        },
        // You are next! Don't do this a Friday afternoon!
        "prod": {
            "enabled": true
        }
    }
}

Once all environment has been rolled out it is possible to update the global targetRevision and remove the chartRevision from each environment

"source": {
        "repoURL": "https://kyverno.github.io/kyverno",
        "targetRevision": "2.8.0"
    },
    "destination": {
        "namespace": "kyverno"
    },
    "clusters": {
        "dev": {
            "enabled": true
        },
        "uat": {
            "enabled": true
        },
        "prod": {
            "enabled": true
        }
    }
}

Experiment value files

Usually it is not allowed to perform changes without opening a Pull Request. However this can be quite annoying when experimenting.

It is possible by creating a branch in which developer will have push permission reference it in the environment:

{
    "source": {
        "repoURL": "https://kyverno.github.io/kyverno",
        "targetRevision": "2.7.0"
    },
    "destination": {
        "namespace": "kyverno"
    },
    "clusters": {
        "dev": {
            "enabled": true,
            "valuesRevision": "feature/new-values"
        },
        "uat": {
            "enabled": true
        },
        "prod": {
            "enabled": true
        }
    }
}

Once all environment has been rolled out you can merge your values on main branch and remove the valuesRevision from each environment

Add Sync options

To add syncOptions such as ServerSideApply it is possible to do it like this:

{
    "source": {
        "repoURL": "https://kyverno.github.io/kyverno",
        "targetRevision": "2.7.0"
    },
    "destination": {
        "namespace": "kyverno"
    },
    "syncPolicy": {
      "syncOptions": {
        "ServerSideApply": true,
        "CreateNamespace": true
      }
    },
    "clusters": {
        "dev": {
            "enabled": true
        },
        "uat": {
            "enabled": true
        },
        "prod": {
            "enabled": true
        }
    }
}

ArgoCD applicationSet

In this section we will create the ApplicationSets that support charts deployment accross the different clusters. I will present two uses cases:

• Case 1: One argocd per environment
• Case 2: One argocd to manage all environment

Generator: One ArgoCD per environment

When using one argocd per environment one ApplicationSet per environment must be created. Here is the example of ApplicationSet for dev environment

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: public-charts
  namespace: argocd
spec:
  goTemplate: true
  generators:
  - git:
      repoURL: https://repo.git
      revision: HEAD
      files:
      # Find all .argocd.json
      - path: 'components/public-charts/**/.argocd.json'
    selector:
      matchLabels:
        # Keep only the one which are activated in dev
        clusters.dev.enabled: 'true' 
  template:
    metadata:
      name: '{{ .path.basename }}'
    spec:
      project: platform
      sources:
        # First source is for value files
        # Default revision is HEAD but can be overriden with clusters.dev.valuesRevision
        - repoURL: https://repo.git
          targetRevision: '{{ dig "clusters" "dev" "valuesRevision" "HEAD" . }}'
          ref: values
        # Second source is for chart
        # Default revision is .source.targetRevision but can be overriden with clusters.dev.chartRevision
        - repoURL: '{{ .source.repoURL }}'
          targetRevision: '{{ dig "clusters" "dev" "chartRevision" .source.targetRevision . }}'
          chart: '{{ default "" .source.chart }}'
          path: '{{ default "" .source.path }}'
          helm:
            valueFiles:
              - $values/{{ .path.path }}/values.yaml
              - $values/{{ .path.path }}/values.dev.yaml
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{ .destination.namespace }}'
      syncPolicy:
        automated:
          prune: true
        syncOptions:
          - ServerSideApply={{ dig "syncPolicy" "syncOptions" "ServerSideApply" "true" . }}
          - CreateNamespace={{ dig "syncPolicy" "syncOptions" "ServerSideApply" "true" . }}

Generator: One ArgoCD to manage all environment

When using one argocd for all anvironment you need only one ApplicationSet. Here is the example of ApplicationSet. It is a bit more complex so I recommend you to take a look above to first understand a basic use case.

Let’s say we have 3 clusters (dev, uat and prod). You need to have 3 Cluster Secrets:

apiVersion: v1
kind: Secret
metadata:
  name: dev-cluster
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: dev
  server: https://mycluster.dev.company.com
  config: |
     ...

apiVersion: v1
kind: Secret
metadata:
  name: uat-cluster
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: uat
  server: https://mycluster.uat.company.com
  config: |
     ...

apiVersion: v1
kind: Secret
metadata:
  name: prod-cluster
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: prod
  server: https://mycluster.prod.company.com
  config: |
     ...

Link: https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#clusters

A note: The clusters in your .argocd.json must correspond to the data field “name” in your cluster secret.

Example if your cluster name is called “user-acceptance”

apiVersion: v1
kind: Secret
metadata:
  name: user-acceptance-cluster
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: user-acceptance
  server: https://mycluster.user-acceptance.company.com
  config: |
     ...

Then you argocd.json must contain a cluster “user-acceptance”:

"source": {
        "repoURL": "https://kyverno.github.io/kyverno",
        "targetRevision": "2.8.0"
    },
    "destination": {
        "namespace": "kyverno"
    },
    "clusters": {
        "dev": {
            "enabled": true
        },
        "user-acceptance": {
            "enabled": true
        },
        "prod": {
            "enabled": true
        }
    }
}

Here is the ApplicationSet:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: public-charts
  namespace: argocd
spec:
  goTemplate: true
  generators:
  # Create a matrix generator that will install all chart in all clusters
  - matrix:
        generators:
            # Get all clusters
          - clusters: {}
          - git:
              repoURL: https://repo.git
              revision: HEAD
              files:
              # Find all .argocd.json
                - path: 'components/**/.argocd.json'
              values:
                # Add enabled field if clusters.${name}.enabled = true
                # name comes from the cluster generator
                enabled: '{{ dig "clusters" .name "enabled" false . }}'
      selector:
        matchLabels:
          # Keep only the one which field enabled=true
          enabled: "true"
  template:
    metadata:
      name: '{{ .path.basename }}'
    spec:
      project: platform
      sources:
        # First source is for value files
        # Default revision is HEAD but can be overriden with clusters.${name}.valuesRevision
        - repoURL: https://repo.git
          targetRevision: '{{ dig "clusters" .name "valuesRevision" "HEAD" . }}'
          ref: values
        # Second source is for chart
        # Default revision is .source.targetRevision but can be overriden with clusters.${name}.chartRevision
        - repoURL: '{{ .source.repoURL }}'
          targetRevision: '{{ dig "clusters" .name "chartRevision" .source.targetRevision . }}'
          chart: '{{ default "" .source.chart }}'
          path: '{{ default "" .source.path }}'
          helm:
            valueFiles:
              - $values/{{ .path.path }}/values.yaml
              - $values/{{ .path.path }}/values.{{ .name }}.yaml
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{ .destination.namespace }}'
      syncPolicy:
        automated:
          prune: true
        syncOptions:
          - ServerSideApply={{ dig "syncPolicy" "syncOptions" "ServerSideApply" "true" . }}
          - CreateNamespace={{ dig "syncPolicy" "syncOptions" "ServerSideApply" "true" . }}

Note: I did not experimented this one, so some adaptations to the ApplicationSet may be required :)

Notes

SyncPolicy

As of today it is not possible to template syncPolicy in the ApplicationSet. Therefore the global syncPolicy as well as the overriden syncPolicy is currently not working.

However I am currently working on this feature. I will update this article as soon as it is available

Link: https://github.com/argoproj/argo-cd/pull/14893

Disclaimer

There is multiple methods to manage different versions of a chart accross different environments. This article is exposing one of theses. In also uses a beta feature (Multiple Source). I recommend you to experiment this in a development environment and validate if this is a solution that fit your use case