Keeping AMIs current with an Amazon EC2 Image Builder
While running containerized workloads and leveraging AWS’ serverless offerings are the wave of the future, sometimes it is necessary to run workloads using good, old-fashioned EC2 instances. The challenge that comes with running your own instances is keeping them patched. AWS Systems Manager Patch Manager does a fantastic job of patching long-running instances, but what about instances that are part of an Auto Scaling group? To solution for this use case, we turned to EC2 Image Builder.
Before we get to the solution, it is important to understand our desired end state. We have four AWS workload accounts (beta, sandbox, preprod, and prod) which each needed to run the same special software on Amazon Linux EC2 instances in an Auto Scaling group. We wanted to maintain one KMS-encrypted AMI which could be shared with each account. Additionally, we wanted a new, patched image to be built each month. Then, at our leisure, it could be deployed into each account, starting with beta. Also, we didn’t want to click any buttons in the AWS Console.
It may sound like a tall order, but with Amazon EC2 Image Builder, Terraform, KMS, and the Instance Refresh feature of Auto Scaling groups, we were all set.
We have a separate account named “central” which creates a nice separation from the workload accounts, so we decided to use that to host the AMI and KMS key.
Step 1: Configure KMS keys
Since the same AMI will be shared with multiple accounts and since it’s a requirement to encrypt it, we needed to create a KMS key and grant access and usage to it for the workload accounts.
Step 2: Configure a Custom AMI
We created an instance from an AWS-provided Amazon Linux 2 AMI. We configured it according to our requirements and also installed the required software on it.
Step 3: Create EC2 Image Builder Pipeline
This step is where a bulk of the work takes place. We created an EC2 Image Builder pipeline which spins up an instance from the AMI we created, runs “yum update,” cuts a new AMI, and shares with with the workload accounts. It does this once a month, according to the schedule we specified.
Step 4: Deploy the new AMI
Just because a new AMI (complete with all the latest updates and patches) has been created and shared doesn’t mean it is in use. There is a variable for each workload workspace which specifies the AMI to use for the EC2 instances in the Auto Scaling group. To deploy the new AMI in a workload account, we modify the variable to contain the new AMI ID. Then we run Terraform which updates the Launch Template used by the Auto Scaling group and also kicks off an Instance Refresh. We do this first in beta, then we work our way up and hit prod last.
Hope this helps should you have a similar use case!