Solved: GitOps with ArgoCD and Kustomize for multi-region, multi-account microservices
TL;DR:
- One gitops repo that contains all deployment files per env per region
- ArgoCD per kubernetes cluster (per env per region)
Preface
ArgoCD is a declarative, GitOps continuous delivery leading solution for Kubernetes.
Kustomize is a native kubernetes configuration management.
We chose kustomize because it’s a simple configuration management for yaml files which we already used and because it supports ad-hoc changes so we don’t need to plan in advance.
In this blog post I will explain how to deploy applications on kubernetes in multi AWS accounts/regions.
In our use case we have a restriction for the deployment configuration: traffic in each region is isolated (security constraint) so we decided to deploy argocd per account/region and it looks as in the picture above.
Note: I didn’t mention secrets in this blog post on purpose. Secrets management is out of scope of this blog and deserves its own post.
Deployment Repo
We created one gitops deployment repository that includes the deployments of all our regions.
The directory structure looks like that:
.
├── base
│ ├── microservice1
│ └── microservice2
└── staging
└── eu-west-1
├── applicationset
│ └── generator
└── team1
├── microservice1
│ └── overlays
├── microservice2
│ └── overlays
└── global
We have a base directory for each app that includes all defaults of our deployment and then we have a folder for each env/region.
In this example we have staging/eu-west-1 which includes the following:
- Applicationset — An argocd object which generates multiple applications from a template. In this template we will configure the apps that are in the team’s folder:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: team1
spec:
generators:
— list:
elements:
— app: global
— app: microservice1
— app: microservice2
template:
metadata:
name: ‘{{app}}’
annotations:
notifications.argoproj.io/subscribe.on-sync-succeeded.teams: channelName
notifications.argoproj.io/subscribe.on-sync-failed.teams: channelName
notifications.argoproj.io/subscribe.on-health-degraded.teams: channelName
spec:
project: default
source:
repoURL: https://gitlab.local/team1/k8s-deployment-argocd.git
targetRevision: HEAD
path: “stg/eu-west-1/team1/{{app}}"
destination:
server: https://kubernetes.default.svc
namespace: team1
syncPolicy:
automated:
prune: true
- team1/global — Is an argocd app that contains our global configmap. We will reference this configmap from all our deployments and use it as a regional variables repository.
- team1/[service] — Is a kustomize app that includes our base of [service] which includes a deployment, configmap and cronjobs for a specific microservice and overlays folder to make changes that are specific to this region.
Summary
ArgoCD architecture as suggested here using Kustomize, works great for multi-cluster, multi-region, multi-account cloud deployments. I hope you have gained some insights as well.
Best,
Yossi Nachum