Building your Gold AMI’s using Packer
What is an AMI?
An Amazon Machine Image (AMI) is a master image for the creation of virtual servers in AWS environment. The machine images are like templates that are configured with an operating system and other software, which determines the user’s operating system. AMI types are categorized according to region, operating system, system architecture(32- or 64-bit), launch permissions and whether they are backed by Amazon EBS or backed by the instance store.
Each AMI includes the following:
- A template for the instance’s root device volume
- Set up permissions that control which Amazon Web Services (AWS) accounts can use the machine images to spin up an instance
- The block devices that specify the root device volume to attach to the EC2 instance once it’s launched.
What is Golden AMI?
Golden Amazon Machine Image is an AMI that you standardise through configuration, consistent security patching, and hardening. It also contains agents you approve for logging, security, performance monitoring and tools necessary to perform various tasks etc. You can build and deploy golden AMIs in your environment, but the AMIs quickly become dated as new vulnerabilities are discovered.
As a security best practice you need to perform routine vulnerability assessments of your golden AMIs to identify if newly found vulnerabilities apply to them. If you identify a vulnerability, you can update your golden AMIs with the appropriate security patches, test the AMIs, and deploy the patched AMIs in your environment.
Advantages
Having a golden AMI saves our time in installing the pre-requisite software configuration and ensures that you have a consistent image that is approved for use by the organization. It also ensures that the machine image that you are using is free from all vulnerabilities and is aligned with security standards.
Below diagram illustrates the high-level workflow of golden AMI pipeline
How to build the golden AMI?
There are various tools available in the market for building the golden AMI’s and some of the prominent one’s are
Hashicorp’s Packer
AWS Image Builder
If we compare these two tools, Packer stands out in terms of below features :
- Packer allows you to build images for multiple platforms like AWS, Azure, GCP and Docker etc using same configuration files.
- Packer is simple in terms of logging and debugging as the output is clearly shown in the command-line when the image is built.
- Packer uses JSON files to build the components which are simple commands and can be easily integrated with CI/CD pipelines.
- Packer has built-in support for using various configuration management tools like Ansible, Chef and Puppet along with Shell and PowerShell script support for installing software.
- Packer is distributed as a lightweight binary and it is controlled using CLI.
The Packer template is divided mainly into three components
User Variables
User variables allow your templates to be further configured with variables from the command-line, environment variables, vault, or files. This lets you parameterize your templates so that you can keep secret tokens, environment-specific data, and other types of information out of your templates. This increases the portability of the template.
Builders
Builders are responsible for creating machines and generating images from them for various platforms. For example, there are separate builders for EC2, VMware, VirtualBox, etc. Packer comes with many builders by default, and can also be extended to add new builders
Provisioners
Provisioners are responsible for installing and configuring softwares on the running machines prior to turning them into machine images. Packer provides in-built support for different kinds of provisioners mainly Shell, File, Poweshell, Ansible, Chef and Puppet etc.
Building the golden AMI
Let’s get our hands dirty by building an AMI which will have AWS Inspector Agent pre-installed. Then we will configure AWS Inspector to scan one of the instances launched from this AMI and find out the list of CIS(Center for Internet Security) Recommendations, CVE’s(Common Vulnerabilities and Exposures) on this instance. We can then fix all the security recommendations using a bash script or an ansible playbook and execute them as part of Packer provisioners so that our golden AMI will be aligned with security standards.
First install Packer by following the steps mentioned below
Since we are building the AMI in AWS account we will make use of “amazon-ebs” builder from Packer. If you want to go for instance-store backed AMI in AWS, you can choose the “amazon-instance” builder.
Below is the packer_build.json template file which we are going to use for building the golden AMI. Packer will first launch a temporary instance based on the configuration provided in the builders section and then install the required softwares/packages from provisioners section and create an AMI out of it.
As you can see, I have defined the AWS credentials, vpc region, instance type and ssh username in variables section. These values are in turn derived from packer_vars.json file where we can define any other required variables.
In the Builders section, I’m making use of filters for VPC, Subnet and Security Group based on the Name tag. Packer will search the resources matching the tags and get their ID’s. Also in this example, we are using Ubuntu AMI and we are filtering the latest version of AMI using the “source_ami_filter” option.
This is an added advantage of Packer over other tools, where we can filter the resources based on tags within the template file. So in cases where your infrastructure is destroyed and re-created by terraform, your packer builder template will not be affected.
We are using a Packer function called “isotime” to generate a unique name(add timestamp suffix to AMI name) every time we build a new one.
By default the AMI’s you create will be private. If you want to share the AMI’s with other accounts you can make use of the “ami_users” option in packer.
In the Provisioners section, we are adding a script to install the Inspector Agent.
echo "*******Installing AWS Inspector Agent********"curl -o "/tmp/inspector_install" https://inspector-agent.amazonaws.com/linux/latest/installsudo bash /tmp/inspector_installsudo /etc/init.d/awsagent startrm -rf /tmp/inspector_install
If you want to validate your packer_build.json file, you can use the below command
packer validate packer_build.jsonBefore proceeding to the next step, make sure that the AWS user you are using for packer-builder has minimal set permissions for Packer to work. You can find the list of required permissions here
Once the validation is successful and the build template is free from any syntax errors, you can build the AMI using below command
packer build -var-file=packer_vars.json packer_build.jsonLet’s see how packer builds the AMI
You can see from the above output, packer is getting the VPC, subnet and security group id’s from the tags and it is creating a temporary keypair and associating it with the instance. Then it installs the AWS Inspector Agent.
This is the final output where Packer publishes the build artifacts (AMI ID’s) and Packer deletes all the temporary stuff which has been created (keypairs, instances etc) so you need not worry about those.
Now we have an AMI with AWS Inspector Agent pre-installed.
Scanning the AMI for Security Vulnerabilities
Let’s configure AWS Inspector to scan for security vulnerabilities in the golden AMI. For this we are going to launch an instance with our new AMI.
When choosing the AMI while launching the instance, you can check for your new AMI in “My AMI’s” section as shown below
Once the instance is up and running, we need to tag it with a specific name because AWS Inspector picks up the instance for scan based on tags. You will also have an option to scan all instances in the account.
The main components of AWS Inspector are
Assessment Target
It defines what instances to be scanned. You can include all instances or use tags to specify particular instances.
Assessment Template
It contains a reference to the Inspector Assessment target and set of AWS managed rule packages to evaluate
- Security Best Practices-1.0
- Common Vulnerabilities and Exposures-1.1
- CIS Operating System Security Configuration Benchmarks-1.0
- Network Reachability-1.1
You can specify the duration of the scan and add SNS topic to notify the events. Also you can schedule a recurring assessment for specific number of days.
Assessment Run
It contains the list of assessment runs performed on targets.
After creating the Assessment target and template, we will initiate an Assessment Run which will scan our instance for vulnerabilities
Inspector has reported 103 findings for our instance. You can view a detailed report by clicking the “Download Report” button under “Reports” column.
Each finding will have a severity tag, description of the finding and recommendation to fix it. All the CIS Recommendations need to be fixed manually and you can create your own custom shell script or an ansible playbook. Configure that as a new provisioner in packer, so that next time when packer builds an AMI it applies all these recommendations and it will be free from any vulnerabilities.
You can add a new provisioner as below
{"type": "shell","execute_command": "sudo -u root /bin/bash -c '{{.Path}}'","script": "./scripts/Ubuntu_CIS_Recommendations.sh"},
This is only first step of building the AMI pipeline. As i mentioned in the beginning, we need to scan the AMI regularly to check if it has any new vulnerabilities, patch/fix them, build new ones and decommission the old/unused AMI’s.
A golden AMI pipeline should have the following
- A portable build template with a set of provisioners for installing required software
- Automated tests to test the newly built AMI
- Scan the new AMI using AWS Inspector to check for any new vulnerabilities
- Decommission old AMI’s after specific retention period
- Notify teams if any of these steps fail
References
