Part II: Using GitOps with ArgoCD to Automate Azure DevOps Agent Deployment to EKS

In this article, we will continue from where we left off in the previous article.

It’s Argo time! To deploy Azure DevOps agents to EKS using the solution provided here, you must install ArgoCD on your cluster. While there are many ways to deploy it, we won’t dive into the specifics here. However, you can refer to this simple guide as a starting point:

argo-helm/charts/argo-cd at main · argoproj/argo-helm

In addition to ArgoCD, we’ll be using several other tools, including:

• Helm Library Charts
• KEDA
• External Secrets Operator

By the end of the process, we’ll have an Azure DevOps agents running on our EKS, which will be connected to a dedicated agent pool in Azure DevOps.

Connecting our CI with our ArgoCD

Our main goal is to do GitOps whenever a new agent version is updated on one of our agent’s Helm charts.

To make the CI/CD as automatic as possible, there are several steps in our current Docker build pipeline that takes the new Docker tag version and updates it in the agent Helm chart values file.

Once ArgoCD identifies a change (reconciliation is every 3 minutes by default), it will change the state of our agent’s root application (App of Apps) to “Out of Sync,” which will allow us to click on Sync and update all the agents to the latest version. This also can be done automatically, and auto-sync can be enabled, but each use-case and its needs

First, Create a personal access token (PAT) in Azure DevOps.

Use personal access tokens - Azure DevOps

Our agent will need the following environment variable to be passed as a secret from Kubernetes:

• AZP_TOKEN: The personal access token (PAT) used for the agent to authenticate with the Azure DevOps organization. This is a secret and will be stored in our secret manager.

When it comes to what level of permissions the token needs to have, the principle of least privilege is recommended, which means granting only the necessary permissions to perform the required actions. Just to remind you, these recommendations are general guidelines, and you should assess your specific use case to determine which permissions are needed.

Here are the permissions that require more than “Read” access, along with some recommendations:

• Agent Pools: Read & manage
• Build: Read & execute
• Code: Read & write
• Connected server: Connected server
• Deployment Groups: Read & manage
• Environment: Read & manage
• Packaging: Read & write
• Pipeline Resources: Use
• Pull Request Threads: Read & write
• Service Connections: Read & query
• Test Management: Read & write
• Tokens: Read & manage
• Wiki: Read & write

It’s important to remember that Personal Access Tokens are a form of credential that grants access to your Azure DevOps account or organization. Thus, they are secrets.
The environment variables that are not secrets will be passed via the ArgoCD application using the agent Helm chart.

I will store PAT as a secret in my AWS Secret Manager for this article. But it can be stored in whatever secrets providers you are using, as long as it is accessible to the Kubernetes cluster.

Let’s start configuring our agents before we deploy them using ArgoCD.

The agent configuration can be found in the eks-azdo-agents branch, which is in the azdo-agents-gitops repository.

To deploy the agents, we need to make sure the values files for both app-of-apps that we have, are correctly configured.

The following environment variables are not secrets and will be configured directly from the agent values file:

• AZP_URL: The URL of the Azure DevOps organization. (i.e. https://dev.azure.com/<organization>). This is not a secret and will be passed via the Helm chart.
• AZP_AGENT_NAME: The name of the agent. If this variable is not set, the hostname will be used as the agent’s name. (the Pod’s name)
• AZP_POOL: The name of the agent pool to use. The “Default” pool will be used if this variable is not set.

Before we continue, you need to ensure you have a dedicated agent pool in your Azure DevOps account. KEDA will listen to its queue.

Create and manage agent pools - Azure Pipelines

External Secrets Operator permissions

Before deploying our agent, we must ensure the External Secrets Operator can fetch secrets from AWS Secret Manager by creating a proper IAM Role.

The following code exemplifies the permissions it needs and the trust relationship policy.

You can find instructions about how to create an IAM role in the link below:

Creating a role to delegate permissions to an IAM user

IAM Role Permissions example:

Trust relationship policy:

For this, you’ll have to provide the OIDC provider of your EKS.

Note: Keep your IAM Role ARN, we’ll need to configure it in the tools App of Apps values file.

Installing ArgoCD and deploying our agent

Since this article is not about installing ArgoCD, I won’t dive into it and will assume you already have it installed.

If you don’t, I suggest installing it via Helm chart and later setting ArgoCD to manage itself via an independent application

You can find instructions for installing ArgoCD in the link below.

Installation - Argo CD - Declarative GitOps CD for Kubernetes

So how do we set up and run our agent via GitOps?

First, we need to connect our GitOps repository (azdo-agents-gitops) to ArgoCD, by adding it as a repository, and there are several ways to do it:

• Using the UI
• Declaratively using a K8s Secret YAML file
• Imperatively via the ArgoCD CLI

Each way has its pros and cons, but we won’t dive into them in this article.

I will add the repository via the UI to keep it simple, but I encourage you to work as much GitOps as possible and store the repositories on your Git.

Declarative Setup - Argo CD - Declarative GitOps CD for Kubernetes

As for the SSH private keys, you can use Sealed Secrets to encrypt them and, thus, not have them exposed in your Git.

Prerequisites for connecting Azure DevOps Git Repository:

• Login with a read only permissions user account to Azure DevOps
• Create a private key for ArgoCD and store it in a secret vault
• Add the public key to Azure DevOps on your read only user account

Use SSH key authentication - Azure Repos

To connect the GitOps repository:

1. Log in to the ArgoCD UI and navigate to the “Settings” page.
2. Click on the “Repositories” tab.
3. Click the “Connect repo using SSH” button.
4. In the “Name” field, provide a name for the repository.

5. In the “Project” field, select the project to which the repository belongs.

6. In the “Repository URL” field, provide the SSH URL for the repository you want to connect.

7. In the “SSH private key data” field, paste in the contents of your SSH private key.

8. Click the “Connect” button to connect the repository to ArgoCD.

The outcome should be a connected repository:

Deploying Azure DevOps agent to our EKS via ArgoCD

And now, to the fun part, make some Argo magic.

We have our two App of Apps root applications ready to be deployed, but before we deploy them, we need to ensure the order in which they will be deployed.

We need KEDA and ESO (External Secrets Operator) to be deployed before the Agents because they depend on the following:

• An already fetched Kubernetes secrets from AWS Secret Manager by ESO
• ScaledObject resource that is part of KEDA APIs

Hence, we will remember that the first root application that needs to be synced is the tools app-of-apps.

Note: there is a way of making all of that being done automatically, but it’s out of this article scope, and I’ll write an article about it.

To deploy our app-of-apps:

1. Open your terminal and open your cloned GitOps repository (from you Azure Git)
2. Git switch to your cluster branch, which contains ArgoCD YAML files.
3. Make sure you have kubectl installed and a connection to your EKS
4. Set your Kubernetes cluster context to the cluster you want to use
5. Create the tools app-of-apps on the namespace that argocd is installed on, usually argocd.

helm template tools/ | kubectl apply -f - -n argocd

Once finished, open your ArgoCD UI and see that External Secrets Operator and KEDA were deployed and the root application that manages them.

This type of installation of ArgoCD applications via kubectl should happen only the first time you want to create the App of Apps, after it, the applications state will be managed by the Git.

After clicking SYNC to the tools-root:

6. Repeat the previous step, but instead of creating the tools app-of-apps, create the agents app-of-apps.

helm template agents/ | kubectl apply -f - -n argocd

Once the root application is created, we’ll need to sync it so that it will make the relevant AppProjects as well.

After syncing:

In the ArgoCD UI, filter the projects to show only the agent’s project:

Now, our agents are deployed and managed via ArgoCD in our managed Kubernetes (EKS).

In Azure DevOps, let’s see if our agents are registered to our Agents pool that we’ve created:

So far, we’ve covered the following topics:

• Building and deploying Azure DevOps agent Docker images to ECR
• Deploying Azure DevOps agents to EKS via Azure DevOps pipelines and ArgoCD

The following is a flow chart of the final architecture as it would be:

But before we continue to the next chapter, here is a question that needs to be asked: do you think these two App-of-Apps should exist? If there is a dependency between the two of them, and currently there is no native support for putting waves on ArgoCD applications.. maybe it’s better to use just regular apps for the tools?

Let me know what you think in the comments below :)

Next Chapter: Building Docker images in Kubernetes with Kaniko and Azure DevOps

Part III: Building and pushing a Docker Image Using Kaniko to ECR