Turbocharge ArgoCD with App of Apps Pattern and Kustomized Helm
Well folks, it’s been a few months now since we’ve started on the ArgoCD journey together, and it’s been a wild ride. But we’re not done yet!
If you’ve been following along at home, we’ve gone from setting up ArgoCD and Tekton on Kubernetes, to creating a Kubernetes-native build and release pipeline with Tekton and ArgoCD, to configuring SSO with Active Directory on ArgoCD.
Application Deployments Revisited
So what’s next? Well, we did a simple app deployment with ArgoCD when we did the Tekton + ArgoCD example, which was a good start.
As you may recall, ArgoCD lets us define an
Application resource, which is responsible for orchestrating the deployment of an application manifest to the target Kubernetes cluster.
NOTE: When I say “application manifest”, I mean all the YAML definitions that make your microservice run in Kubernetes. For example,
Serviceresource definition and a
Application definition points to a folder where the manifests are located. Out of the box, ArgoCD supports plain ’ole YAML, Kustomize, Helm, and Jsonnet. (Note: Ksonnet is deprecated as of the time of this writing.)
That’s all well and good, but what about when we have to do something a bit more complicated? Like deploy a bunch of related microservices??
Stick around, because I’ll explain all of that, AND I’ll have a short tutorial at the end that you can try out for yourself.
To follow along in the tutorial, you’ll need to have a Kubernetes cluster with ArgoCD installed. If you need help installing ArgoCD on Kubernetes, DON’T PANIC! Check out my blog post on how to install ArgoCD on an existing Kubernetes cluster on a TLS-enabled Ambassador API Gateway.
The App of Apps Pattern
Since we’re using ArgoCD, we need to create an
Application definition for each microservice being deployed. But, since they’re part of an app bundle, it would be nice if we had a way to ArgoCD know that.
NOTE: I refer to “app bundle” as a group of related microservices.
This is where the App of Apps Pattern comes into play. And lucky for us, it’s not rocket science!
Put plainly, the App of Apps Pattern lets us define a root ArgoCD
Application (Root App). Rather than point to an application manifest, the Root App points to a folder which contains the
Application YAML definition for each microservice that’s point of the app bundle (Child Apps). Each microservice’s
Application YAML then points to a directory containing the application manifests.
The diagram above depicts four component microservices which make up the Guestbook App.
The Root App’s definition looks like this:
Looking at a sample Child App definition, we see something like this:
In the sample Child App above, when we look at Line 14, it tells ArgoCD to look in the
helm-guestbook folder of the source repo for Kubernetes manifests. In this case, it’s the folder where a Helm Chart is defined. The folder could’ve easily contained a Kustomization or plain old Kubernetes YAML files.
One thing I’d also like to point out is Line 11, which states that the destination cluster is
dev-cluster. This is different from Line 11 in the Root App definition, whose destination value is
in-cluster. This is because the Root app must be deployed to the cluster in which ArgoCD is installed (i.e.
in-cluster). The Child App, however, will most likely be deployed to a different cluster — like your DEV, QA, or PROD cluster.
In case you’re wondering how to find the cluster name, simply run this command:
argocd cluster add
The output will show something like this:
CURRENT NAME CLUSTER SERVER
* dev-cluster dev-cluster https://<some_ip>
qa-cluster qa-cluster https://<some_other_ip>
The value from the
NAME field in the output above is what you need.
App of Apps Best Practices
It’s worth noting a few best practices around the App of Apps pattern.
1- Create a project for each app bundle
ArgoCD projects provide a logical grouping of applications. They can also be used to:
- Define trusted Git repos (so you can’t deploy apps pointing to just any old git repo and potentially wreak havoc)
- Define what types of resources can be created on a cluster (maybe you don’t want someone to be creating
- Define who has access to the project, and therefore, to the
Applicationsin the project.
- Restrict what clusters you can deploy an app to. Our good ‘ole separation of concerns at work here.
ArgoCD projects are defined using special ArgoCD resource called
You can check out a sample ArgoCD
AppProject definition below:
The above is a very simple definition and therefore doesn’t take advantage of all of the
AppProject goodies listed above, but it gives you a good start.
2- Create environment-specific app bundles and projects
So, we agree that we should create an
AppProject per application bundle. Let’s take it a step further and create an
AppProject per application bundle per environment. When you think about it, it makes sense. You’ll have to create a separate application bundle definition for each target environment anyway, since you’d need to target different clusters each time you deploy to a different environment.
And yes, you can for sure get fancy and do some templating to create your
3- Keep Application definitions in a separate repo
My personal preference is to have the ArgoCD
AppProject definitions in a separate repo from the source code repo.
Why? Because chances are, the team managing application deployment to Kubernetes via ArgoCD is different from the team writing the actual application code.
If you’re still not sure right now, don’t worry. You’ll get a better idea of the setup when we get to the example below.
First, why would I want to do Kustomized Helm? Well, Helm is great at some things, and not so great at other things. Same goes for Kustomize.
Helm is great for templating.
Kustomize is great for:
- Applying common configs to a set of YAML files at once (e.g.labels, namespaces, annotations)
- Overriding values by applying selective changes to YAML files
Fortunately, enabling Kustomized Helm on ArgoCD is not hard at all. All we need to do is update the
argocd-cm.yml definition by adding an entry for a new plugin, and call it
kustomized-helm, like so:
Okay…so what the heck does this do?
We’re telling ArgoCD to run the command on Line 19 if we choose to use the newly-created plugin called
helm template ../../helm_base — name-template $ARGOCD_APP_NAME — include-crds > ../../helm_base/all.yml && kustomize build
The above command is doing the following:
1- Render the chart template locally
To do this, we use the
helm template command, which, per the above command, saves the output to
We’re telling ArgoCD to look for the
Chart.yaml in the
helm_base folder. This means that our repo must have a
helm_base folder in order for this to work.
2- Apply a Kustomization to
To do this, Kustomize expects a
kustomization.yml. In this case, the
kustomization.yml must be in the
Here’s what our sample
kustomization.yml would look like:
Now, if you’ve seen ArgoCD’s sample argocd-cm.yml for configuring Kustomized Helm, you’d notice right away that mine differs slightly, in that I “force” you to have this
helm_base folder in your repo. The reason why I do this is because I want to make use of Kustomize’s overlays, which is perfect if I want to use different values for different environments to which I’m deploying my app. I basically end up with a file structure like this:
And if I want to define an ArgoCD Application to use the
dev overlay, I end up with something like this:
Note that on Line 14, we set the path to
kustomized_helm/overlays/dev. Referring back to our file structure diagram above, ArgoCD will look in
kustomized_helm/overlays/dev for a
kustomization.yml file, which looks something like this:
Line 5 refers to our
helm_base folder, which is where our base
kustomization.yml is located (the one that references the resource
So when we run
argocd app sync, the process looks something like this:
Okay…so now that we know what the heck this mysterious plugin is doing, let’s apply it to our ArgoCD cluster so that it’s available:
kubectl apply -f argocd-cm.yml
Now that we have this plugin defined, how do we use it? To do so, you simply refer to the plugin in your ArgoCD
kustomized-helm is the name that we gave the plugin that we defined above in Line 13 in
Application YAML looks like this:
Now that you’ve got a good idea on how the App of Apps Pattern works, let’s put it into practice, shall we?
I’ve basically got the following setup:
This repo contains the YAML manifests which define the following for DEV, QA, and PROD environments:
- The ArgoCD project in which the app will reside
- The ArgoCD root app
- The ArgoCD child apps (2048-game and Guestbook)
Our good ‘ole friend the 2048 game makes a return appearance! Here I’ll be demonstrating what a Kustomized Helm deployment looks like.
This is based on one of the examples from the ArgoCD example repo, and I just have it here to show what it looks like to deploy two “apps” with the App of Apps pattern. Don’t expect this app to function on its own.
To run the example below, you’ll need the following:
- A Kubernetes cluster
- ArgoCD running on the Kubernetes cluster
If you need help installing ArgoCD on Kubernetes, DON’T PANIC! Check out my blog post on how to install ArgoCD on an existing Kubernetes cluster on a TLS-enabled Ambassador API Gateway.
1- Login to ArgoCD
To login to ArgoCD, replace the values in
<…> with your own values, and run the snippet below:
export ARGOCD_SERVER=<argocd_server>argocd login $ARGOCD_SERVER --username $ARGOCD_USERNAME --password $ARGOCD_PASSWORD
2- Register the repos with ArgoCD
We need to register our repos ArgoCD, so let’s add the 3 repos:
export GIT_TOKEN=<git_personal_access_token>argocd repo add https://github.com/d0-labs/argocd-app-of-apps-parent --username git --password $GIT_TOKENargocd repo add https://github.com/d0-labs/argocd-app-of-apps-child-2048-game --username git --password $GIT_TOKENargocd repo add https://github.com/d0-labs/argocd-app-of-apps-child-guestbook --username git --password $GIT_TOKEN
You can now list your repos:
argocd repo list
Which will give you an output that looks something like this:
Or if you want to check it out in the GUI, go into the ArgoCD console and click on the gear icon on the left-hand pane, and then click on Repositories:
3- Create the Dev Project
Now that we’ve got our repos registered, let’s create our project. Again, we’re creating a project to encapsulate our application. We’re also sticking with our best practice of having a different project per environment.
NOTE: For this tutorial, we’re creating all resources for the DEV environment.
kubectl apply -f argocd/projects/project-dev.yml
Hurray! We’ve created a new project, which you can see by going into the ArgoCD console and clicking on the gear icon on the left-hand pane, and then clicking on Projects:
Or, if you’re a CLI-lover like me, you can run:
argocd proj list
Which gives you something like this:
4- Create the Root App
We are now ready to create our Root App in ArgoCD. We can do it with
kubectl, since, an ArgoCD
Application is a (custom) Kubernetes resource:
kubectl apply -f argocd/root-app-dev.yml
As soon as we create our app, we can see it on the ArgoCD console:
But we’re not done yet. You’ll notice that the status says
If we click on that tile, we’ll see this:
Okay…so ArgoCD recognizes the Root App and its two children! It means that it knows that there are
Application manifests for the two children, and also even found them, but they haven’t actually been created in Kubernetes. In order to do that, we must run an
argocd app sync.
Let’s go ahead and do that now.
5- Sync the Root App and its children
First, let’s sync the Root App:
argocd app sync root-appbundle-app-dev
We immediately see that the child Applications have been created!
But they haven’t been synced. So if we drill into each of these apps, we see this:
So let’s sync the two child apps with this command:
argocd app sync -l app.kubernetes.io/instance=root-appbundle-app-dev
Note how we don’t even list the individual child apps by name! That’s because we use the special ArgoCD label,
app.kubernetes.io/instance, and give it the value
root-appbundle-dev, the name of our Root App! That means that if your app consists of 10 apps, you’ll only ever need the above two sync commands (one for the Root App, and one for the Children)!
If everything syncs correctly, you’ll see something like this:
And drilling into each Child App, we see this:
One of the coolest things about using the App of Apps pattern is that when you delete the Root App, by default it deletes the Child Apps and all of their resources, in one fell swoop! That includes namespaces too. So you end up with a very clean delete.
To delete our Root App and all of its children:
argocd app delete root-appbundle-app-dev
More on the
argocd app delete command here.
Note that the above command does NOT delete projects and repos. Those need to be done separately.
Deleting our repos from ArgoCD:
argocd repo rm https://github.com/d0-labs/argocd-app-of-apps-parentargocd repo rm https://github.com/d0-labs/argocd-app-of-apps-child-2048-gameargocd repo rm https://github.com/d0-labs/argocd-app-of-apps-child-guestbook
Deleting our project from ArgoCD:
argocd proj delete appbundle-project-dev
You can’t delete a repo unless you delete all apps associated with that repo first.
Similarly, you can’t delete a project until you delete all apps associated with that project first. If your project has associated repos, you must delete those before you can delete the repo.
It’s worth noting an ArgoCD app deployment gotcha that has caused me a bit of grief in the past. Hopefully this section will save you some hair-pulling.
Sometimes you might notice that when you deploy an application, it will show up as
Healthy, but the Sync Status is
Unknown. Kind of like this:
This is not good. It means that something went caca, and most likely your
Application definition has an error.
When we click on the tile above to drill in, we see this:
We definitely see an error (see above, highlighted), because we don’t see the Child Apps. This means that ArgoCD didn’t find the manifests in version control for them. Something is definitely stanky.
We should be seeing something to the diagram below.
You can click on the
Error to see what’s up. It reveals this:
You can also see what’s going on by tapping into our good buddy,
kubectl describe command for our app,
kubectl describe application -n argocd root-appbundle-app-dev
NOTE: All ArgoCD applications are created in the
Which gives us output that may look something like this:
In our case, it’s pointing to some sort of authentication issue. Upon closer look, the repo we’re referring to in the spec,
d0-argocd-app-of-apps-parent is wrong. We don’t have such a repo registered. We registered
d0- prefix). Sunofa…
After your fix the error, I recommend that you just nuke the
Application from ArgoCD either via the GUI or
argocd app delete, and recreate it again using
That’s a wrap!
Okay…we’ve learned a lot today!
We learned that:
- The App of Apps pattern is great for bundling related applications together.
- Creating a separate project in ArgoCD for each App of Apps bundle, per target Kubernetes environment (DEV, QA, PROD) is good.
- Kustomize + Helm make a hell of a tag-team!
- When you need to nuke your App of Apps app bundle, deleting the Root App in ArgoCD by default does a cascade delete, which cleans up all Child App resources, including namespaces, so you don’t lose your mind doing cleanup.
- App deletion in ArgoCD does not nukify projects and repos.
And because you’ve put up with yet another long-ass blog post, here’s a picture of a cuddly little silkie chicken. (Silkies, by the way, are the cutest chickens EVER.)
Peace, love, and code.
One more thing…
In the spirit of DevOpsyness, it would be cool if we had a cleaner way of creating all of the App of Apps manifests and Kustomized Helm templates discussed here, without having to do it manually, right?
I’m currently working on a follow-up blog post to discuss just that. In the meantime, check out our newly-open-sourced tool that does a bunch of that ArgoCD magic. And stay tuned for the follow-up blog post!
Other stories in my ArgoCD journey
Want to know more on how we got here? Check out the other stories in my ArgoCD journey below!
Installing Ambassador, ArgoCD, and Tekton on Kubernetes
Configuring your Kubernetes cluster for Kubernetes-native build and release with Tekton and ArgoCD
Using Tekton and ArgoCD to Set Up a Kubernetes-Native Build & Release Pipeline
A Tekton and ArgoCD primer and step-by-step guide for setting up and running build & release workflows with Tekton and…