Deploy “N” Preview Pull Requests to “N” clusters

Published in

The Zeals Tech Blog

5 min readDec 20, 2021

alias PR=Pull Request, CD = Continuous Delivery, CI = Continuous Integration

Did you feel the need to make infrastructure based on a Pull Request upstreamed by a developer or an engineer to test their changes and remove the infrastructure bits after they have tested their changes?

Did you ever think that for a Pull Request made in Github there should be a corresponding infrastructure that should spin up in accordance with the pull request?
Do you feel the need of a temporary customised namespace and resources made on the runtime for developers to test their changes against it?

If the answer to above question is affirmative, let’s get this done today !!

Background

At Zeals, We strongly felt the need of spinning up infrastructure bits based on Pull Request in its separate custom namespace to enable our engineers to verify there changes before going to merge those and eventually deploying to cluster and making it part of our application. This solution could solve some major problems for us

A preview/temporary environment based on pull request, so that quality of pull request can be evaluated without it being getting merged into our repository
Every pull request should have its own namespace in which it is tested and then all resources including the namespace are removed once the PR is merged or Closed thus being cost effective too.

We were already using ArgoCD for our CD pipelines and it was working perfectly well. We wanted dynamic PR-based infrastructure in place. I followed some different paths to get this accomplished. I explored Argo Bot but it was limited in the functionality it provides. On a side note JenkinsX already lets you spin up preview environment for the PRs generated.

Existing Approaches:

There were few blogs that I read about before diving into my own implementation of things. I wanted a way without scripting, native to ArgoCD.

I found Codefresh blog really interesting that also tells about the preview environments but it also sort of lacked an ArgoCD way of doing things
There is also a very well written Container Solutions blog but this involved a lot of scripting that I did not want
There is a brilliant video out there, explaining how we can use ArgoCD for preview environments, but that too was not using latest ArgoCD concepts and there was some extra tools involved
I found ArgoCD PR generator which settled well with our infrastructure and there was no need to use any other tooling or extra scripting

Before kicking it off I am going to tersely define some terms/concepts that would help you getting good grip on this article (this is for novice readers, if you are an expert please skip and jump to Bootstrapping section)

Gitops: GitOps works by using Git as a single source of truth for declarative infrastructure and applications.
Kubectl: The Kubernetes command-line tool, kubectl, allows you to run commands against Kubernetes clusters
Kind: kind is a tool built for running local Kubernetes clusters using Docker containers as nodes.Its quite useful for creating a Kubernetes environment for local development, QA, or CI/CD.
Note: kind requires Docker to be installed
Kustomize: Kustomize is a template-free declarative approach to Kubernetes configuration management and customisation.

ArgoCD

Argo CD is a declarative, GitOps continuous delivery tool, which allows developers to define and control deployment of Kubernetes application resources from within their existing Git workflow

ApplicationSet Controller

ApplicationSet is the evolution of App of apps pattern and added more flexibility to it for dealing with a wide range of use cases

The ApplicationSet controller is a sub-project of Argo CD which adds Application automation, and seeks to improve multi-cluster support and cluster multi-tenant.

Generator
Generators are responsible for generating parameters, which are then rendered into the template: fields of the ApplicationSet resource.

Git Generator
The Git generator allows you to create Applications based on files within a Git repository, or based on the directory structure of a Git repository.
Pull Request Generator
The Pull Request generator uses the API of an SCMaaS provider (eg GitHub) to automatically discover open pull requests within an repository.
Matrix Generator
The Matrix generator may be used to combine the generated parameters of two separate generators.

I have used the following combination of generators here

Matrix = Git + Pull Request

As of writing this blog, Argo applicationSet supports the following list of generators. Depending on the use-case , there is a freedom of combining generators in order to achieve your desired result.

Generators = List + Cluster + Git + Merge + Pull Request + SCM Provider + Cluster Decision + Matrix
                              OR
Generator 1 = Matrix(List + Pull Request + Git )
Generator 2 = Matrix(Pull Request + Cluster Decision + Git)
........

Directory Structure

Using the power of GitOps concepts and kustomize tooling, our directory structure looks like

Cluster-configs: contains the dev cluster definition
N number of cluster files can be added if you want to deploy N number of clusters e.g qa, prod, stg
Pipeline: This contains the applicationset and project for application managed by ArgoCD

├── clusters-config
│   └── dev-cluster.yaml
└── preview-app
├── base
│   ├── deployment.yaml
│   ├── kustomization.yaml
│   └── service.yaml
├── pipeline
│   ├── preview-applicationset.yaml
│   └── preview-project.yaml
└── sandbox
└── kustomization.yaml

Bootstrapping the environment

Pre-reqs

There are some requirements to get this done (Mostly the readers would have all this setup, maybe with a bit of difference, but I am going to touch these and move forward). I am going to write down the version with which the current setup is tested.

Kustomize (v4.4.1)
Kind (v0.11.1)
ArgoCD (client: v2.2.1)
Kubectl (client: v1.23.1)
k8s (v1.12.1)
Note: The described toolset will be already there but if not you can easily get the components installed.

GitOps Engine Start….

Now that you have all the pieces setup its time to setup the backend

First of all we need to create a cluster using kind

2. We have Cluster up and running now we are going to install ArgoCD & ArgoCD ApplicationSet

3. Now we need to create ApplicationSet and project in ArgoCD. You need to run the following commands to get it done.

This is currently pointing to my GH repo, you can use your own GH repo or fork mine too, and change the internal code to your own repo URLs. You may need to change repoURL, owner, repo and labels (Please note there are multiple repoURL in file that you must replace)

4. Moving ahead we need to point the generators to the repo we want it to watch for dynamically creating preview environment based on our PR.

5. The following code block explains the name of the application which will be dynamically generated. The generated app-name will be made by adding the prefix preview, branch name, PR number and cluster name to target cluster. The app name will give you complete description of the PR owner and the deployed cluster. I have added comments to various sections so the code is self explanatory