How To: Enable History and Rollback for Argo CD Application Set with Multiple Sources

Intro

We will showcase how we were able to enable History and Rollback for Argo CD Application Set with Multiple Sources feature. We will showcase the demands that we were asked to meet and how we overcame some of the difficulties that we encountered on the way to meet those demands.

Developers’ Application Sets requirements

The Developers asked us to meet two requirements that the Application Set must meet for it to be implemented.

1. Support 3 diffident repositories

• “Chart repo” — an artifactory where the chart resides.
• “Values repo” — the repo that the helm values will be passed from.
• “Manifest repo” — yaml manifests in this repo will represent the Argo CD Applications. For example:

# manifest repo - directory structure example 
applications/
 ├── guy-test.yaml
 ├── guy-test2.yaml
 └── guy-test3.yaml

Note — the repo names above will be used to represent them in the rest of this article.

2. The ability to use History and Rollback from Argo CD UI

On first glance this sounds like a very simple thing to do. But as we will go along, we will see that we have encountered some problems and demonstrate how we were able to solve.

What Will be Discussed

• What is Application Set
• Application Set Git Generator
• How to overcome Application Set Sources limitation

Version:

Argo CD 2.6.7

What is Application Set

Application Set helps us to automate the generation of Argo CD Applications.

The ApplicationSet controller is a part of Argo CD adds Application automation, and seeks to improve multi-cluster support and cluster multitenant support within Argo CD. Argo CD Applications may be templated from multiple different sources, including from Git or Argo CD’s own defined cluster list. The controller automatically generates Argo CD Applications based on the contents of a new ApplicationSet Custom Resource (CR).

Application Set Git Generator

What are Argo CD Generators

Generators are responsible for generating parameters, which are then rendered into the template: fields of the ApplicationSet resource.

Git Generator

The Git generator contains two subtypes: the Git directory generator, and Git file generator, for my use-case I will use the second type — Git file generator.

Git File Generator— The Git file generator generates parameters using the contents of JSON/YAML files found within a specified repository.

We will use the Git file generator to generate our applications. The Git Generator below will look for yaml files. Starting from the repo root directory, it will look for files that match the following path applications/*.yaml .

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: main-application-generator
  namespace: argo
spec:
  goTemplate: true
  generators:
  - git:
      repoURL: '<manifest_repo_url>'
      revision: HEAD
      files:
      - path: applications/*.yaml

For example, applications/guy-test.yaml manifest will have the following key value structure from which the Git Generator would generate the values for spec.template for our Applications.

# applications/guy-test.yaml
name: guy-test
chart:
  name: my-chart
  version: 1.0.0
namespace: default
image: <image_url>:<image_tag>
source:
  repoURL: '<values_repo_url>'
  path: '<path_to_values.yaml>'
  targetRevision: '<git_branch>'

We can use these key values in order to generate Argo CD Applications dynamically with different set of values depending on the file that represents each application.

# the applicatipon set manifest that will be deployed to the clsuter
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: main-application-generator
  namespace: argo
spec:
  goTemplate: true
  generators:
  - git:
      repoURL: '<manifest_repo_url>'
      revision: HEAD
      files:
      - path: applications/*.yaml
  template:
    metadata:
      name: '{{.name}}'
      namespace: argo
      finalizers:
        - resources-finalizer.argocd.argoproj.io
    spec:
      project: default
      syncPolicy:
        automated:
          selfHeal: true
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{.namespace}}'
      sources:
        - chart: '{{.chart.name}}'
          repoURL: '<chart_artifactory_url>'
          targetRevision: '{{.chart.version}}'
          helm:
            valueFiles:
            - $values/path/to/values/file/values.yaml # weakest enforcement - this file not have to contain the image url and tag
            - $manifest/applications/'{{.name}}'.yaml # the manifest that represents the application will also contain the image that will be used
            - $manifest/values/'{{.namespace}}'/values.yml # values that will be enforced on all applications under this name space
            - $manifest/values/values.yml # strongest enforcement - values the will be enforced on all applications
        - repoURL: '{{.source.repoURL}}' # this repo url will be defined from applications/guy-test.yaml
          targetRevision: '{{.source.targetRevision}}'
          ref: values # indicate the currect refrence
        - repoURL: '<manifest_repo_url>' # some values will be set from manifest_repo
          targetRevision: HEAD
          ref: manifest

At first, we thought that this will be a simple task using Application Set with the Git Generator, as just shown. This seemed to answer all the requirements.

Even though this method worked and we were able to generate applications as expected, we had 2 limitations that we had to overcome:

1. How to define specific ignoreDifferences per Application
2. When using the multiple sources feature, the HISTORY AND ROLLBACK feature is disabled with a tooltip saying it is not supported yet. Which is a known issue in Argo CD.

https://github.com/argoproj/argo-cd/issues/12580

How to overcome Application Set Sources limitation

Define specific ignoreDifferences per Application

We were able to overcome this by using jqPathExpressions as well as using an activation flag from the chart. Here is an example of ignoreDifferences for autoscaling minReplicas:

# application set spec
spec:
  template:
    spec:
      ignoreDifferences:
      - group: autoscaling
        kind: HorizontalPodAutoscaler
        jqPathExpressions:
        - select(.metadata.annotations.ignoreDifferencesMinReplica=="true") | .spec.minReplicas

In this example only if the activation flag set by an annotations is true Argo CD will ignore minReplicas. The annotations can be added to HorizontalPodAutoscaler resource, and be set to True or False using helm values. This way we have control over when we want to ignoreDifferences.

HISTORY AND ROLLBACK feature is disabled when using multiple sources feature

As mentioned in the beginning one of the requirements was to allow HISTORY AND ROLLBACK feature for rollback purposes.
Because this is not a supported feature when using sources we had to be crafty and see how we could do the same thing using the source feature, and not sources.

STEP 1: Changed the sources to source and set the Values repo as the source. We did not change the Git Generator so we could still use all the key values that we set in my application file.

# application set spec
spec:
generators:
  - git:
      repoURL: '<manifest_repo_url>'
      revision: HEAD
      files:
      - path: applications/*.yaml
  template:
    ... # same as example about until source field
    source:
      repoURL: '{{.source.repoURL}}'
      path: '{{.source.path}}' # path example values/app1
      targetRevision: '{{.source.targetRevision}}'
      helm:
        valueFiles:
        - values.yaml # path will look for values.yaml under values/app1

Using this configuration we could not set the values that were configured from different files as before. So we had to set some values directly to the Application Set, combining valueFiles and values. The order matters, and and the following example values will be enforced over valueFiles.

# application set spec
spec:
generators:
  - git:
      repoURL: '<manifest_repo_url>'
      revision: HEAD
      files:
      - path: applications/*.yaml
  template:
    ... # same as example about until source field
    source:
      repoURL: '{{.source.repoURL}}'
      path: '{{.source.path}}' # path example values/app1
      targetRevision: '{{.source.targetRevision}}'
      helm:
        valueFiles:
        - values.yaml # path will look for values.yaml under values/app1
        values: |
          image:
            repository: {{.image.url}}
            tag: {{.image.tag}}
          cloud: <cloud_provider>
          region: <region>
          namespace: {{.namespace}}
          cluster: <cluster_name>
          environment: <env>

In this example, we can see that we can still set the image and namespace dynamically from the file that represents our application, appplications/guy-test.yaml .

STEP 2: Setting the chart artifactory from the values repo.

Under my Values repo I had to create Chart.yaml at the same path where the values.yaml was. The Chart.yaml will be defined with a dependency for the actual chart that I want to use:

values/
└── app1/
   ├── Chart.yaml
   └── values.yaml

apiVersion: v2
name: guy-test
type: application
version: 1.0.0
dependencies:
  - name: my-chart
    version: "1.0.0"
    repository: "<artifactory_url>"

Because now we are using helm dependencies, we will have to set our values accordingly. Now Chat.yaml is where we will define the chart and version our application will be using. In the beginning this was set from appplications/guy-test.yaml .

We did not remove chart.name from appplications/guy-test.yaml , we did that so we could use the chart name dynamically depending on the chart that we will be using. As you can see because we are now using a dependency chart we have to specify the contexts for where the values will be passed to our dependency chart as follows in the example below.

Not shown here — but do not forget to add the chart name to values.yaml under the values repo as well.

# application set
    ... # same as the example above until source field
    source:
      repoURL: '{{.source.repoURL}}'
      path: '{{.source.path}}' # path example values/app1
      targetRevision: '{{.source.targetRevision}}'
      helm:
        valueFiles:
        - values.yaml # path will look for values.yaml under values/app1
        values: |
          {{.chart.name}}: # specifying chart dependency’s context
            image:
              repository: {{.image.url}}
              tag: {{.image.tag}}
            cloud: <cloud_provider>
            region: <region>
            namespace: {{.namespace}}
            cluster: <cluster_name>
            environment: <env>

STEP 3: Add argocd.argoproj.io/hook: Skip to replicate Disable Auto-Sync. When creating Argo CD Applications using Application Set we cannot configure syncPolicy manually.

When using HISTORY AND ROLLBACK we manually override the syncPolicy that was set, usually from Enable Auto-Sync to Disable Auto-Sync. But with Application Set whenever we try to manually override the Application syncPolicy the Application Set will reset it as was configured in the Application Set manifest.

# application set
spec:
  template:
    spec:
      syncPolicy:
        automated:
          selfHeal: true

We can overcome this by using Argo CD skip hook. Skip indicates to Argo CD to skip the manifest that have this annotation. By adding this annotation to all helm chart manifests, with an activation flag, we can simulate a state where all the manifests of the chart will be skipped even if there is a new commit to the git repo.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: some-manifest
  annotations:
    {{- if .Values.historyAndRollback }}
    argocd.argoproj.io/hook: skip
    {{- end }}

So when .values.historyAndRollback is set to true we could use the history and rollback feature to roll back to the desired version, using the UI HISTORY AND ROLLBACK feature. When we are done and ready to move back to sync mode, all we have to do is set .values.historyAndRollback back to false.

Summary

With some crafty configurations we were able to create an Application Set that interacts with 3 deferent sources, even though only one source was set for the templated Applications.

This craftiness allowed us to create a path towards enabling HISTORY AND ROLLBACK feature for our Applications.

References:

• https://argo-cd.readthedocs.io/en/stable/user-guide/application-set/
• https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators/
• https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-Git/
• https://github.com/argoproj/argo-cd/issues/12580