Demystifying GitOps -Bootstrapping Argo CD

9 min readMar 8, 2022

In the first part of this series I have talked about the general concepts of GitOps including a little bit of GitOps history, definition of GitOps, flagship GitOps tools. With this post I want to start diving into the details. You can find the links for the completed ones of the series at the bottom.

Before Codefresh team has implemented Argo CD Autopilot as an open source project and hand over to Argo community, Argo CD didn’t have any opinionated tool to bootstrap Argo CD. I mean, it has Argo CD cli but Argo CD cli does not deploy an opinionated GitOps setup. It solves different problems. There are best practices applied by the community and patterns implemented around these experiences like App of Apps(which I will explain below). Those who are familiar with Flux can easily imagine what a bootstrapper cli is and how that is helpful since Flux uses Flux bootstrap cli to deploy an opinionated GitOps setup. I plan to talk about Flux and Flux bootstrap cli in detail in another post. This post is about Argo CD, so let me reconcile this post to its focus.

In this post I will demonstrate to bootstrap a self managed Argo CD in three different ways. Firstly, I will use App Of Apps pattern which is born to setup Argo CD with its best practices. Then, I will use ApplicationSets which bring several generators to apply App Of Apps pattern easily. Finally, I will use Argo CD Autopilot to demonstrate an opinionated Argo CD setup. Before diving into bootstrapping Argo CD, let me explain basic Application Definition to some extend.

Application CRD

Application in Argo CD is a CRD which manages different kind of resource definitions like deployment, secret, configmap and many. It can also manage different type of resource packages like helm, kustomize, raw manifest files. A basic application resource definition is below.

apiVersion: argoproj.io/v1alpha1 
kind: Application 
metadata:   
  name: guestbook   
  namespace: argocd 
spec:   
  project: default   
  source:     
    repoURL: https://github.com/argoproj/argocd-example-apps.git 
    targetRevision: HEAD     
    path: guestbook   
  destination:     
    server: https://kubernetes.default.svc     
    namespace: guestbook
  syncPolicy:    
    automated:      
      selfHeal: true      
      prune: true

If we examine above application definition, beyond the basic stuff, it includes a source property containing git repository url which includes resources to be applied. In this repository, latest commit under the guestbook directory will be applied to the destination. Destination contains a kubernetes cluster reference which points to the default cluster in Argo CD operator. Basically resources will be applied to guestbook namespace in the same kubernetes cluster where Argo CD is installed.

App Of Apps

Application CRD seems to be complete in itself but it doesn’t address some challenges. Those challenges are, how can Argo CD instance be managed by Argo CD itself(some sort of chicken and egg problem)? How can I manage lots of applications, clusters and their applicaiton definitions by using Argo CD? These challenges have been addressed with the emergence of App of Apps pattern. App of Apps can be described as one application definition manages all other application definitions. These application definitions manage the applications(resources) we want to deploy to our kubernetes cluster or clusters.

A basic implementation of App Of Apps pattern can be seen in my Github repository. It contains a parentapplication folder including the application definition for the child application definitions which are committed to childapplications folder. Childapplications folder is planned to contain application definitions to be managed by this Argo CD instance. It only contains Argo CD application definition for now which solves the chicken and egg problem for Argo CD bootstrapping. By this way a self managed Argo CD setup can be deployed. Application definitions for any other application can be committed to this folder. This Katacoda Scenario is implemented to show the flow. If you want to change Argo CD setup after installing(increasing the replica count of Argo CD controller), you can just copy files from my Github Repository and add it to your own repository. Then run this scenario with your own Github Repository address. After installing you can change the Argo CD pod replication count for example, commit and push the changes. Don’t forget to wait a few minutes.

Parent Application Managing Argo CD Application

Argo CD Application Managing Argo CD(Only Pods and Deployments Shown)

ApplicationSets

Argo CD ApplicationSets is an implementation of App Of Apps pattern with support of a kubernetes operator(ApplicationSet controller). This operator has to be deployed alongside Argo CD Operator. The addition of this operator brings new CRDs to the cluster where Argo CD is deployed. These CRDs are abstractions over Application Definition CRD. We can also define it as a template generator since this operator generates Application Definitions from ApplicationSet templates. It started with three main generators, List Generator, Cluster Generator and Git Generator. Then new generators were added, Matrix Generator, Merge Generator, SCM Generator, Pull Request Generator, Cluster Decision Resource Generator. You can find the link for details about all generators in the references section below.

Lots of scenarios are covered with these generators. For example, SCM provider generator can be used to automatically deploy resource definitions in newly added repositories to the cluster without a single line of configuration or a Pull Request Generator can be used to create ephemeral environments for each feature branch pull request. I plan to work on a separate post for all of these generators. In this post I will use Git Generator to bootstrap Argo CD.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: gitopsdemo
  namespace: argocd
spec:
  generators:
  - git:
      repoURL: https://github.com/a1tan/argoapplicationsets.git
      revision: HEAD
      directories:
      - path: managementstack/*
  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/a1tan/argoapplicationsets.git
        targetRevision: HEAD
        path: '{{path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{path.basename}}'
      syncPolicy:
          automated:
              allowEmpty: true
              prune: true
              selfHeal: true

Above ApplicationSet definition includes an array on the generators property. For the first and only generator of this definition there is a link to a git repository and a path definition. This “managementstack/*” wildcard path means create an application definition for each folder under managementstack parent folder. If that parent folder is visited, two sub folders can be seen. One folder contains argo cd deployment and other folder contains applicationset definition itself.

By this way two Argo CD applications are deployed. These applications manage Argo CD and applicationset definition itself. This is very much similar to previous App Of Apps pattern deployment. As It can easily be seen, ApplicationSets need much less configuration code. One thing here must be noted, in a production setup a private repository is needed in addition to these and of course a secret to read all definitions from private repository. This Github Repository and Katacoda Scenario includes steps to be followed and resources.

Parent Application Managing itself and argocd application definition both

argocd Application Definition Managing Argo CD

Argo CD Autopilot

Argo CD autopilot is a cli tool to bootstrap self managed Argo CD on a Kubernetes cluster and creates a Git repository as the single source of truth. It is still under active development. You can check the status from roadmap link on the references.

It defines an opinionated structure for managing applications through Argo CD. Argo CD Autopilot is not a requirement for experienced Argo CD users but especially new users can benefit from experiences of the community by using this tool. When you bootstrap your Argo CD setup with autopilot you get a structured private git repository, opinionated folders to put your own application resources and logical grouping of applications named projects, self managed Argo CD controllers including ApplicationSet controller and a secret to pull desired state from git repository.

Steps to install(With Github):

Configure kubeconfig file with your cluster information and set current context correctly
Create a PAT on your Github account
export GIT_TOKEN=$(PAT)
export GIT_REPO=https://github.com/owner/repository
run “argocd-autopilot repo bootstrap”

And wait until cli completes its work, so simple.

Feel free to use this Katacoda Scenario but don’t forget to delete PAT you have created after you are done. You can check the result of running this scenario in this Github respository. It contains three folders:

“bootstrap” folder contains all bootstrapping resources as shown below. “root.yaml” file contains application definition for projects folder. Projects folder is for the logical grouping of applications. This projects folder can be used to separate different environments like dev, test and prod.

“cluster-resources.yaml” is an applicationset CRD managing cluster-resources folder by pointing the cluster-resources/in-cluster.json file. It uses Argo CD “Git Generator:File” sub generator and manages the “argocd” namespace in the kubernetes cluster.

“argo-cd.yaml” is an application definition to self manage Argo CD itself.

Argo CD Autopilot has also commands to add new projects and applications. It creates and commits these changes to Github Repository and desired state is applied by Argo CD operator.

You can see the result of bootstrapping Argo CD with autopilot below.

autopilot-bootstrap Application Definition to manage whole cluster(An opinionated implementation of App Of Apps)

argo-cd application definition to self manage Argo CD

Pros And Cons

With raw Argo CD you have to handle some problems as I mentioned. App Of Apps pattern was designed to solve these problems and it did. First implementations of this pattern did not add any new CRDs to the system. Because of this it needed to be implemented with many configuration files. Argo community defined this problem really well and implemented ApplicationSet CRDs. With the implementation of these CRD based generators we gained ability to solve the problems with much less configuration and effort. However, it can easily be said that, ApplicationSet is not a silver bullet either. As we see the result of Autopilot bootstrapping on the third part, Application CRDs and ApplicationSet CRDs can be used together.

Argo CD Autopilot tries to solve different problems in addition to the solved problems by App of Apps. It puts an opinionated way to bootstrap Argo CD setup by including App Of Apps pattern. It also uses ApplicationSet CRDs.

On the pros side, it deploys an Argo CD setup adhering to best practices and security concerns. It also defines a folder structure to logically separate your applications(e.g. for environmental separation). We haven’t covered this separation in other demos and we need a similar structure for production use.

On the cons side, we may not want the folder structure it creates or we may want more customization but it may not be still there for the current version. It is also under active development at the time of this writing and some basic features are missing.

Conclusion

In this post, Argo CD is bootstrapped in three different ways. In the first part, raw Application Definitons are used to bootstrap the Argo CD with App of Apps pattern, in the second part newer ApplicationSet operator of Argo CD is used and for the final part new opinionated bootstrapping tool Argo CD Autopilot is used. As I mentioned before if you are experienced and know what to do, you can bootstrap Argo CD with ApplicationSet CRDs and App Of Apps pattern but if you are a beginner Argo CD Autopilot is definitely helping. There are also some planned improvements on Autopilot cli like handling secrets in a secure way in Git Repositories and many. It is better to follow changes and jump on the train when available features solve our problems.