Part I: How I Built and Deployed Azure DevOps Agents to ECR
In this article, we will continue from where we left off in the previous article.
To get started, you must create two new repositories in your Azure Repos Git that will hold the Agent Dockerfiles, script provided by Microsoft and the CI pipeline YAML file, and ArgoCD manifests.
To start, you can clone the following repositories from my personal GitHub, which contains all that is needed (azdo-docker-agents, azdo-agents-gitops), and use them or push them to your repositories.
While the azdo-docker-agents repository has a standard structure for repositories that build Docker images, this is not the case in azdo-agents-gitops, which is more unique (and one of many possible ways) that I chose to implement GitOps.
The main branch will act as the GitOps branch and will be monitored by ArgoCD.
It will have the following:
- Library Helm Chart for the Agents contains what is needed to deploy an agent on Kubernetes.
- Agent’s Helm charts that use the Library Chart as a dependency
On a branch representing our ArgoCD applications, called eks-azdo-agents, we’ll have two App-of-Apps:
(you can read more about App of Apps here)
agents/
- An applications folder for the ArgoCD agent’s applications, configurations, and secrets
- An eso-secrets folder containing the External Secrets Operator CRDs, which in turn contain the secrets to be fetched from the AWS secret manager.
- A configuration folder for the argo-cd-cm, argocd-rbac-cm, cluster-role-agent, and cluster-role-binding-agent. The cluster role and role binding are necessary to allow the agent to create a Kaniko pod during the pipeline for building and deploying Docker images. We will talk about it later.
- A projects folder that contains the ArgoCD appProjects
- The Agents root application that manages the rest of the applications
tools/
- An applications folder for the External Secrets Operator (ESO), ESO Secrets and KEDA
- A projects folder that contains the ArgoCD appProjects for configurations and tools.
- The tools root application that manages the rest of the applications
Once the repositories are created, we can create the CI pipeline to build and deploy the Docker images to ECR.
The CI pipeline which exists in the azdo-docker-agents repository will perform the following actions:
- Build Docker images for generic use and Python 3.8.
- Use the generic Docker image as a base for other framework agents such as Python, Java, and Dotnet.
- Update all relevant Dockerfiles with the recent whenever a new base image is built.
However, before creating the pipeline, we must secure our AWS credentials (Access and Secret access keys) by saving them in a secret management system.
For this article, we will use Azure DevOps Library Groups and seal the variables that contain the credentials so that they are only accessible to the pipeline to consume.
The variable names can be changed to suit your needs.
An essential part of the pipeline is to increment the Docker image version each build and update the
the VERSION file (both in the root and the pyhon38 directory), the Dockerfile base image of specific frameworks, and the Docker image tag in the Helm chart of the Agent, which is in the azdo-agents-gitops repository using a commit & push.
To push the commit to the Agents and GitOps repositories, you must provide the necessary permissions for the Azure DevOps builder user.
Follow these steps: (on both repositories)
- Select your repository from your Azure DevOps project.
- Click on the “Branches” tab and select the main branch.
- Click on the three-dot menu and select “Branch security.”
- Under “Users and groups,” select the builder user.
- Ensure the following permissions are enabled in the “Permissions” section.
After providing the necessary permissions, you can create the pipeline and point it to the “azdo-docker-agents” repository. To create the pipeline, follow the instructions in this link:
And use the azure-pipelines.yaml which can be found in the agents Docker repository.
Once you finish reviewing the pipeline YAML, click on “Save.”
Before running the pipeline, you need to make some adjustments to the variables YAML file, which is in the folder .azure-devops/azure-pipeline-variables.yaml
You can play with the values as needed, but the main change will be in the AWS Account ID and the region.
Now decided to build our first generic Agent, which will also be the base image for the Python agent.
Please create a corresponding ECR repository so the image can be pushed. Ensure that you have a corresponding ECR repository already created so that the image can
If you decided to store your credentials in Azure DevOps library groups, after triggering the pipeline, it will request permission to access it.
Finally, you should see a screen that shows the pipeline has been completed successfully.
Upon successfully running the pipeline, the Azure DevOps agent Docker image has been deployed to our ECR repository.
Let’s continue to the Python agent. The process is similar. Remember that the repository in ECR needs to exist before the pipeline runs; otherwise, you will get a 404 when the pipeline tries to push the image.
Next Chapter: Automating Azure DevOps Agent Deployment to EKS with ArgoCD and GitOps
