Cloud Custodian: Easy way to explore instances in AWS cloud
Depending on your company guidelines, an older machine can represent a vulnerability. Companies may prefer staying with older, more tested, versions of code, but it’s usually thought that every machine should have the newest operating systems and software versions. But, to update machines you need to identify them first! If you use the AWS console GUI and spend 20 seconds for each instance, you would spend around 3 minutes looking for 10 instances — and more than 5 hours for 1000 instances! So you need an easier and faster way to do it.
Cloud Custodian is a tool that can help you to explore and make changes to the cloud environment in seconds. In this article, we will focus on finding obsolete, older machines by using this tool, so we could later proceed to delete, update or replace them, for instance.
Cloud Custodian is a Python tool that can unify dozens of scripts that organizations use for managing cloud accounts supporting AWS, Azure, and GCP environments. For this example, we are going to work with AWS. For Python, the official documentation recommends version 3.6 or higher.
For the installation process please refer to this link and follow the steps according to your operating system. For Linux systems, we will first create a virtual environment in Python, according to the standards, and then install the c7n package.
python3 –m venv custodian
pip install c7n
You will need to set up write and read IAM permissions to allow Cloud Custodian access. Please check this AWS IAM tutorial. For this example, the IAM role needs read and write permissions on EC2 resources.
To start using Cloud Custodian we need to create a YAML file containing the policy which includes Resource, Filter, and Action. You can combine filters and actions to meet your requirements.
- Resource: This is the type of resource to filter and apply the action (eg EC2).
- Filters: We can specify the target resources using filters based on any criteria like name date or tags for example.
- Actions: It defines what Cloud Custodian will do over the filtered resources.
For more information about policies, please check this link. In this case, we will want to tag old instances. As we are looking for running instances older than 60 days, we name the policy file “ec2-old-dev.yml” and save it into a folder called “policies”.
The “ec2-old-dev.yml” file will filter all the running EC2 resources based on a tag called “Environment” that contains the value “DEV” and is older than 60 days. After those EC2 resources are filtered the policy will create a new tag called “old” with a value of “true”.
With the following command, we can confirm if the policy is accurate and consistent.
custodian validate policies/ec2-old-dev.yaml
custodian.commands:INFO configuration valid:policies/ec2-old-dev.yml
Now that we have the policy file, let’s see how to use it.
When we use the run command, Cloud Custodian will take the resources, filters, and actions as input to translate it into an API Call.
To run the policy we need to specify some parameters.
-s: The path for saving results.
- Policy file: The policy to execute.
- Region: The region for execution.
custodian run \
-s output policies/ec2-old-dev.yml \
--cache-period 0 \
This command will create a folder called “output”, another folder inside called “ec2-old-dev”, and a file called “resources.json” inside of it.
--region=us-west-1: This is the region where the policy will apply
--profile=your-profile: The profile that contains access to the AWS environment.
--cache-period: We can use this option to minimize API Calls. The default option will store the data locally for 15 minutes.
The policy was applied and those running instances that comply with the filters have a new tag called “old” with the value of “true” as we specified in the policy.
We can check the new tag on all those EC2 resources in the AWS console.
Optionally, you can create a report based on the previous command and get all the information you need. The command report needs some parameters.
-s: This is the same parameter from the “run” command. The report command specifies where the “resources.json” file is.
- Policy file: The same policy from the “run” command.
- Region: The region where you ran the previous command.
--no-default-fields: Removes the default fields from the report.
--field: The field that you want to include in your report. Eg.:
custodian report -s /output policies/ec2-old-dev.yml \
You will get a result from the previous command on the terminal. Only the filtered resources and the specified field, in this case only the ImageId.
The result of the command will be
The option “
--field ID=ImageId” will create a column called “ID” and will only provide the information stored on “ImageId”. Also, the option called “
--no-default-fields” will remove the default values from the report and get only the required fields from the “field” parameter.
We can save the output as a CSV file by adding “
> results.csv” at the end of the command.
A Custodian report is useful in case you want to save history or make auditory because you can send that file through email or share it on any communication channel
Now that we have a list of filtered instances according to the specifications, we can decide if we’ll update it or replace it.
For example, we can decide to stop all instances having the tag called “old” with the value of “true”. To achieve that goal we have several options and for this example, we will use AWS CLI with this script
aws ec2 stop-instances --instance-ids $(aws ec2 describe-instances --query ‘Reservations.Instances.InstanceId’ --filters “Name=tag:old,Values=true” --output text) --region=us-west-1 --profile=your-profile
You can also create a Jenkins job that will stop all instances based on tags, please check this link for more information.
Cloud Custodian is a flexible tool that can help you to explore and modify the AWS environment according to your needs.
In this article, you could verify how you can explore the infrastructure and make changes in an easy and fast way.
For more information related to this tool please visit the official documentation.