Automate Code Deployment with AWS EC2 Build Agents for your Azure DevOps Pipelines
Two households, both alike in dignity, combine to create the hybrid cloud CI/CD pipeline of your dreams
We’re huge fans of automation at Hashmap. We believe that your code should be committed often and deployed automatically.
For that reason, we take advantage of Azure DevOps Pipelines whenever we can. For my primary dbt project, I use Pipelines to control all deployment of warehouse code.
However, if you deploy as often as we do, you may find that you’re running out of the free build minutes provided by Azure every month.
To make sure your builds have unlimited runway, this tutorial shows how to use Azure Pipelines to orchestrate builds that run on AWS EC2 Virtual Machines instances. This approach allows for scaling up and out of our build agents to accommodate all the complexity and concurrency we need.
Quick Disclaimer
For this demo, I’ve used the 64-bit x86 Amazon Linux 2 AMI on a t2.micro instance. If you use a different OS or architecture, you may need to modify the setup bash scripts to reference the appropriate package managers and Azure DevOps Pipeline agent setup artifacts.
Build your EC2 Instance
Use any instance size you like, but you’ll need the following:
- public internet access in whichever VPC you use
- allow incoming SSH traffic in your subnet configuration
Make sure you keep track of your SSH key. You’ll need it to connect to your EC2 instance later. Also, do yourself a favor and name and tag your instance.
If you need more details on setting up your EC2 instance, I can suggest this Hashmap Megabytes tutorial that can provide more guidance:
Configure Azure DevOps
These steps are a simplified and easier to follow version of the official build agent instructions. Feel free to refer to the official instructions as needed.
Create a Personal Access Token (PAT) in Azure DevOps in your user settings. Remember this value as you’ll need it later. This is really the only part of this process that sucks. I don’t like that this is the only way for your agents to authenticate with your DevOps instance and I hate that the max life on the token is 1 year. You might want to add a README or something to your repo to remind the team to change this in a year.
In your DevOps project, go to project settings > agent pools and select “Add Pool”. Name this Pool without spaces and remember the name of the pool for later.
Save the following bash script to a file called ec2_build_agent_setup.sh:
#!/bin/bash# install packages
sudo yum update -y
sudo yum install git tar -y# install devops build agent
mkdir azure_devops_build_agent && cd azure_devops_build_agent
curl https://vstsagentpackage.azureedge.net/agent/2.166.4/vsts-agent-linux-x64-2.166.4.tar.gz -O# configure devops build agent
tar zxvf vsts-agent-linux-x64-2.166.4.tar.gz
rm vsts-agent-linux-x64-2.166.4.tar.gz
./config.sh# configure the devops build agent service and run it
sudo ./svc.sh install
sudo ./svc.sh run
Use scp to copy the ec2_build_agent_setup.sh file into your new instance. You will need your EC2 keypair from when you created your EC2 instance to authenticate the copy command. The command will look similar to:
scp -i ~/.ssh/{your_ec2_keypair}.pem ./ec2_build_agent_setup.sh ec2-user@{your_ec2_url_or_public_ip}:~ssh into your EC2 instance. Again, you’ll need that keypair below. The command will look similar to
ssh -i ~/.ssh/{your_ec2_keypair}.pem ec2-user@{your_ec2_url_or_public_ip}Now, run the following in your EC2 instance that you’ve just ssh'd into:
cd # get to your home directory where the setup is
sh ./ec2_build_agent_setup.shThe config wizard that is started by the ec2_build_agent_setup.sh script will require you to provide the following four pieces of information:
- The URL of your DevOps project. It will look like the following:
https://dev.azure.com/{your-organization} - Your PAT from the first step of this guide.
- The name of your agent pool.
- A good, descriptive name for your new pipeline build agent.
The final portion of the script will start the DevOps agent service in your EC2 instance.
You can confirm that all is well with your agent by going to your project settings > agent pools > select your pool >look for the new agent. You should see that it is online.
At this point, you may use the above steps to create more build agents in this same pool if necessary. No need to create a new PAT each time, but it may be more secure if you do.
Using Your New Pool in a Pipeline
It couldn’t be simpler to use your new pool in your Azure DevOps Pipelines.
In your pipeline YAML definition, simply add the following to the top of your YAML file:
# Put this at the top of your yaml
pool: '{name_of_your_new_pool}'
# add the rest of your pipeline definition belowThat’s It!
This approach is all you need to add some jet fuel to your automated builds.
While I used EC2 above, you can make this work on any Linux / Windows / MacOS machine that has public internet access. Simply replace the ec2_build_agent_setup.sh script with similar steps that work on your platform. Check here for detailed instructions for your platform.
Some of My Other Stories
I hope you’ll check out some of my other recent stories also…
Feel free to share on other channels and be sure and keep up with all new content from Hashmap here.
Also, you can catch Randy as a host on Hashmap On Tap, a podcast focused on all things data engineering and the cloud — available on Spotify, Apple, Google, and other popular audio apps.
Randy Pitcher is a Cloud and Data Engineer (and OKC-based Regional Technical Expert) with Hashmap providing Data, Cloud, IoT, and AI/ML solutions and consulting expertise across industries with a group of innovative technologists and domain experts accelerating high-value business outcomes for our customers.
Be sure and connect with Randy on LinkedIn and reach out for more perspectives and insight into accelerating your data-driven business outcomes or to schedule a hands-on workshop to help you go from Zero to Snowflake.
