ArgoCD at scale with ApplicationSet & Go Template

Introduction

In version 2.5 ArgoCD will support Go template for ApplicationSet. We will see in this article how it will help you managing several Applications using a simple ApplicationSet and multiple Configuration files.

Before starting you must be familiar with Argocd. So let’s do a small presentation.

From their website:

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes

Another tool for continuous delivery ??

Usually when you want to perform continous delivery with pipelines such as Azure Devops, Jenkins, GitLab CI etc, you are pushing your changes to the different environments:

ArgoCD helps you continuously deliver (deploy) your code to any Kubernetes environment with a pull mechanism.

How does this work ?

An Application is a Kubernetes Custom Resource which indicates to ArgoCD which Git repository you want to monitor and the targeted revision. If any change happens on this revision, ArgoCD will reflect it into your cluster (sync).

Your repo can contain:

• Kubernetes manifests
• Helm Chart
• Kustomize resources

A small example.

You want to deploy a website composed of a Vuejs Frontend and a NodeJs backend. Your docker images are published to Docker.io and you now want to deploy this application in your Kubernetes Cluster.

In your repository, you may have a manifests folder with:

• Two deployments: One for your UI and one for your Rest API
• Two services: One for your UI and one for your Rest API
• One ingress: for your UI
• Some Configmaps

You just have to define the following Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-website
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: argocd
    server: https://kubernetes.default.svc 
  project: default
  source:
    repoURL: https://github.com/myprofile/my-website.git
    targetRevision: HEAD
    path: manifests/
  syncPolicy:
    automated:
      prune: true
      allowEmpty: true
      selfHeal: true

What will happen ? Argocd will monitor continously your repository my-website on the HEAD revision. Each changes on the HEAD revision will be synched to your cluster.

The bonus:

• You can manage several Subernetes server
• You have a beautiful UI showing you the sync status of your application (the components, logs, events etc)

From ArgoCD website:

For more information I invite you to take a look to ArgoCD documentation https://argo-cd.readthedocs.io/en/stable/

This is an Amazing tool for small projects or big application with a small amount of customer. But what about multiple customers projects ?

ArgoCD at scale with ApplicationSet

Presentation

Let’s say you have one helm chart to deploy your product and you reached the 100s of customer. You want to deploy an instance of your product per customer because you have different configuration from one to another (customisation, external APis etc)

With a standard (push) CD tool you will have to do:

With ArgoCD you could simply create one Application per customer and point to the correct value file. This will avoid having to install your changes on all the stack and have your pipeline waiting for all jobs to complete.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-website-customer-1
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: argocd
    server: https://kubernetes.default.svc 
  project: default
  source:
    path: path/to/values
    repoURL: https://github.com/myprofile/my-website.git
    targetRevision: main
 helm:
   valueFiles:
        - values.customer1.yaml
  syncPolicy:
    automated:
      prune: true
      allowEmpty: true
      selfHeal: true

The problem

Managing 100s of Application Files value Files can be chalenging and error prone.

Here is some maintainability challenges:

• Chart evolution. If it implies changes on the format of the value files (without necessarly impacting the behavior), you will have to reflect theses change on the 100s value files
• Develop UI to configure customers / phases: if you need to integrate the values content in a UI for your Product Owners/QA to view/configure easily customers / phases. You will have to tightly stick the model of your UI to your value format, which will imply changes on your UI each time you want to upgrade your chart to a non backward compatible change.

An easy example:

You want to configure the number of replicas based on the size of your customer.

Initially the number of replicas is located in:

api:
  replicas: 5

Then you introduce an Horizontal Pod Autoscaler to scale up and down based on the traffic of your customer. You just want to have a minimum of 2 replicas for High Availability and the maximum of replicas equals to the initial number of replicas.

so the values become:

api:
  minReplicas: 2
  maxReplicas: 5

You will have to update the 100s of value files with the correct format and also update the UI model for nothing.

ApplicationSet with ConfigFiles

An ApplicationSet is a Kubernetes Custom Resource which is in charge of representing a set of Application based on Generators (list, cluster, git etc).

Here is an example:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: my-website
spec:
  generators:
    - list:
        elements:
          - name: customer1
          - name: customer2
          - name: customer3
  template:
    metadata:
      name: "my-website-{{name}}"
      labels:
        customer: "{{ name }}"
        environment: prod
    spec:
      project: default
      destination:
        namespace: mynamespace
        server: https://kubernetes.default.svc
        project: default
      source:
        helm:
          valueFiles: |
            - values-{{ name }}.yaml
        path: my-website
        repoURL: >-
          https://github.com/myprofile/my-website.git
        targetRevision: main

Based on the above 100s customers example. Instead of having 100 Applications and 100 value files, you will only have one ApplicationSet and 100 value files.

Initially ApplicationSet was using fasttemplate. However this was not really powerful as it did not support conditions, loops etc. So it was not possible to have complex logic.

ApplicationSet now supports Go Template (+sprig functions) which bring all needed feature to have a real logic.

What do you need:

In a https://github.com/myprofile/my-website-configs.git you can host config files with the following tree:

-- customers
---- dev
------ customer-dev-1.json
------ customer-dev-2.json
---- uat
------ customer-test-1.json
------ customer-test-2.json
---- prod
------ customer-1.json
------ customer-2.json
------ customer-3.json

The content of your config file:

}
 "name": "customer1",
 "activateFeature": true,
 "additional": {
  "configuration": "config1"
 },
 "config": [
  "config1",
  "config2",
  "config3"
 ]
{

And finally your ApplicationSet

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: my-website
spec:
  generators:
    generators:
    - git:
        repoURL: https://github.com/myprofile/my-website-configs.git
        revision: HEAD
        files:
          - path: "customers/prod/**.json"
  template:
    metadata:
      name: "my-website-{{ .name }}"
      labels:
        customer: "{{ .name }}"
        environment: prod
    spec:
      project: default
      destination:
        namespace: mynamespace
        server: https://kubernetes.default.svc
        project: default
      source:
        helm:
          values: |
            api:
              replicas: {{ .replicas }}
              {{ - if .activateFeature }}
              feature:
                value: {{ .additional.configuration }}
              {{ - end }}
              configs
              {{- range .config }}
                 - name: {{ . | title | quote }}
              {{- end }}    
        path: my-website
        repoURL: >-
          https://github.com/myprofile/my-website.git
        targetRevision: main

What will happen:

1. Argocd ApplicationSet Controller will detect 3 config files in your repository
2. For each file it will template your Application based on the content of the config file
3. Apply all the Application which will be handled by Argocd Application Controller

With this kind of configuration you have only one ApplicationSet and 100 of config files. This is really useful to handle Applications at scale. You can also upgrade the values format without any impact on the Configuration file.

With the above HPA example:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: my-website
spec:
  generators:
    generators:
    - git:
        repoURL: https://github.com/myprofile/my-website-configs.git
        revision: HEAD
        files:
          - path: "customers/prod/**.json"
  template:
    metadata:
      name: "my-website-{{ .name }}"
      labels:
        customer: "{{ .name }}"
        environment: prod
    spec:
      project: default
      destination:
        namespace: mynamespace
        server: https://kubernetes.default.svc
        project: default
      source:
        helm:
          values: |
            api:
              replicas: {{ .replicas }}
        path: my-website
        repoURL: >-
          https://github.com/myprofile/my-website.git
        targetRevision: main

Then after integrating HPA

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: my-website
spec:
  generators:
    generators:
    - git:
        repoURL: https://github.com/myprofile/my-website-configs.git
        revision: HEAD
        files:
          - path: "customers/prod/**.json"
  template:
    metadata:
      name: "my-website-{{ .name }}"
      labels:
        customer: "{{ .name }}"
        environment: prod
    spec:
      project: default
      destination:
        namespace: mynamespace
        server: https://kubernetes.default.svc
        project: default
      source:
        helm:
          values: |
            api:
              minReplicas: 2
              maxReplicas: {{ .replicas }}
        path: my-website
        repoURL: >-
          https://github.com/myprofile/my-website.git
        targetRevision: main

You can see that we keep the replicas field from your config files and just put it in maxReplicas field (leaving minReplicas to 2).

Conclusion

ArgoCD is really powerful to address continous delivery / deployment however it does not scale when you want to manage 100s of customer. However ApplicationSet with config files helps you addressing such challenges.

Link to my PR: (for Go developers, please be nice I am a Java developer :D) https://github.com/argoproj/argo-cd/pull/10026