Globant
Published in

Globant

Cloud Custodian: Easy way to explore instances in AWS cloud

Depending on your company guidelines, an older machine can represent a vulnerability. Companies may prefer staying with older, more tested, versions of code, but it’s usually thought that every machine should have the newest operating systems and software versions. But, to update machines you need to identify them first! If you use the AWS console GUI and spend 20 seconds for each instance, you would spend around 3 minutes looking for 10 instances — and more than 5 hours for 1000 instances! So you need an easier and faster way to do it.

Cloud Custodian is a tool that can help you to explore and make changes to the cloud environment in seconds. In this article, we will focus on finding obsolete, older machines by using this tool, so we could later proceed to delete, update or replace them, for instance.

Getting started

Cloud Custodian is a Python tool that can unify dozens of scripts that organizations use for managing cloud accounts supporting AWS, Azure, and GCP environments. For this example, we are going to work with AWS. For Python, the official documentation recommends version 3.6 or higher.

For the installation process please refer to this link and follow the steps according to your operating system. For Linux systems, we will first create a virtual environment in Python, according to the standards, and then install the c7n package.

python3 –m venv custodian
source custodian/bin/activate
pip install c7n

You will need to set up write and read IAM permissions to allow Cloud Custodian access. Please check this AWS IAM tutorial. For this example, the IAM role needs read and write permissions on EC2 resources.

IAM Role

The policy

To start using Cloud Custodian we need to create a YAML file containing the policy which includes Resource, Filter, and Action. You can combine filters and actions to meet your requirements.

  • Resource: This is the type of resource to filter and apply the action (eg EC2).
  • Filters: We can specify the target resources using filters based on any criteria like name date or tags for example.
  • Actions: It defines what Cloud Custodian will do over the filtered resources.

For more information about policies, please check this link. In this case, we will want to tag old instances. As we are looking for running instances older than 60 days, we name the policy file “ec2-old-dev.yml” and save it into a folder called “policies”.

Policy example

The “ec2-old-dev.yml” file will filter all the running EC2 resources based on a tag called “Environment” that contains the value “DEV” and is older than 60 days. After those EC2 resources are filtered the policy will create a new tag called “old” with a value of “true”.

With the following command, we can confirm if the policy is accurate and consistent.

custodian validate policies/ec2-old-dev.yaml
custodian.commands:INFO configuration valid:policies/ec2-old-dev.yml

Now that we have the policy file, let’s see how to use it.

Custodian run

When we use the run command, Cloud Custodian will take the resources, filters, and actions as input to translate it into an API Call.

To run the policy we need to specify some parameters.

  • -s: The path for saving results.
  • Policy file: The policy to execute.
  • Region: The region for execution.

Example:

custodian run \
-s output policies/ec2-old-dev.yml \
--cache-period 0 \
--region=us-west-1 \
--profile=your-profile

This command will create a folder called “output”, another folder inside called “ec2-old-dev”, and a file called “resources.json” inside of it.

  • --region=us-west-1: This is the region where the policy will apply
  • --profile=your-profile: The profile that contains access to the AWS environment.
  • --cache-period: We can use this option to minimize API Calls. The default option will store the data locally for 15 minutes.

The policy was applied and those running instances that comply with the filters have a new tag called “old” with the value of “true” as we specified in the policy.

We can check the new tag on all those EC2 resources in the AWS console.

EC2 instance tagged

Custodian report

Optionally, you can create a report based on the previous command and get all the information you need. The command report needs some parameters.

  • -s: This is the same parameter from the “run” command. The report command specifies where the “resources.json” file is.
  • Policy file: The same policy from the “run” command.
  • Region: The region where you ran the previous command.
  • --no-default-fields: Removes the default fields from the report.
  • --field: The field that you want to include in your report. Eg.: --field ID=ImageId

Example:

custodian report -s /output policies/ec2-old-dev.yml \
--region=us-west-1 \
--profile=your-profile \
--no-default-fields \
--field ID=ImageId

You will get a result from the previous command on the terminal. Only the filtered resources and the specified field, in this case only the ImageId.

The result of the command will be

“ID”
“i-AAABBB”
“i-CCCDDD”
“i-EEEFFF”

The option “--field ID=ImageId” will create a column called “ID” and will only provide the information stored on “ImageId”. Also, the option called “--no-default-fields” will remove the default values from the report and get only the required fields from the “field” parameter.

We can save the output as a CSV file by adding “> results.csv” at the end of the command.

A Custodian report is useful in case you want to save history or make auditory because you can send that file through email or share it on any communication channel

Now that we have a list of filtered instances according to the specifications, we can decide if we’ll update it or replace it.

For example, we can decide to stop all instances having the tag called “old” with the value of “true”. To achieve that goal we have several options and for this example, we will use AWS CLI with this script

aws ec2 stop-instances --instance-ids $(aws ec2 describe-instances --query ‘Reservations[].Instances[].InstanceId’ --filters “Name=tag:old,Values=true” --output text) --region=us-west-1 --profile=your-profile

You can also create a Jenkins job that will stop all instances based on tags, please check this link for more information.

Summary

Cloud Custodian is a flexible tool that can help you to explore and modify the AWS environment according to your needs.

In this article, you could verify how you can explore the infrastructure and make changes in an easy and fast way.

For more information related to this tool please visit the official documentation.

--

--

--

Our thoughts as a strategic disruptor in business and cognitive transformation. Visit us at www.globant.com

Recommended from Medium

My top to fight Imposter Syndrome

What Does the Future Hold for Next-Generation Cloud Database Technology in the Cloud Native Era?

That’s how I learnt how to write clean code

Using Azure DevOps Pipelines to Build a Solidity Smart Contract

Yocto Project: more than a build system

How to explore the Open States GraphQL schema

System Stability Assurance for Large Scale Flash Sales

f1Studioz acquires ITJ

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Javier Gimenez

Javier Gimenez

System Engineer. I write about technology. I love learning and sharing

More from Medium

Enable GCP Audit Logs with Terraform

Cloud Pricing Comparison: AWS vs. Azure vs. Google Cloud Platform in 2022

GCP Routing Adventures (Vol. 1)

How I passed the HashiCorp Terraform Certification