How to Boto?

If you have ever interacted with resources in AWS Cloud programmatically, chances are you have already used Boto. Most cloud engineers interface with AWS using either the console (User Interface) or the CLI (Command Line Interface). However, a vast majority of them use AWS SDK in Python called Boto library as part of their automation. In this article we are going to learn about some quick hacks to start using Boto in the most effective way. To start with, let’s look at how to pass credentials to Boto for instantiation.

Ways to pass credentials

When it comes to using Boto, the first thing to keep in mind is how to pass AWS credentials to the library. For someone who is new to Boto, this can be really tricky. There are multiple ways in which you can provide credentials:

1. Set credentials in environment variables
2. Set credentials in AWS credentials file
3. Pass credentials directly in your code

Types of AWS credentials

Now, before we dive into each of those different ways, it is important to understand the different types of AWS credentials. There are broadly two types:

1. AWS keys: These are permanent IAM keys which are long lived until they are disabled or deleted. These are generated by AWS IAM service
2. AWS Session tokens: These are short lived tokens which have a fixed validity and expire after certain duration. These are generated by AWS STS service

You can pass any of the above two types to Boto. The only difference is, when using AWS Session tokens, you will have to additionally specify session key along with access key and secret key.

If you are setting credentials in environment variables:

$ export AWS_ACCESS_KEY_ID=<dummy access key>
$ export AWS_SECRET_ACCESS_KEY=<dummy secret key>
$ export AWS_SESSION_TOKEN=<dummy session token if using AWS Session token>
$ export AWS_DEFAULT_REGION=us-west-2

If you are setting them in AWS credentials file:

$ vim ~/.aws/credentials
[default]
aws_access_key_id = <dummy access key>
aws_secret_access_key = <dummy secret key>
aws_session_token = <dummy session token if using AWS Session token>

And if you are setting them in code itself:

import boto3
session = boto3.Session(
    aws_access_key_id="<dummy access key>",
    aws_secret_access_key="<dummy secret key>",
    region_name="us-west-1"
    aws_session_token='<dummy session token if using AWS Session token>' # (optional... only if using AWS session tokens)
)

Once you instantiate the global session object, you can create a client object for any AWS service like EC2 service below

ec2_client = session.client('ec2')
print(ec2_client.describe_regions())

When calling Boto APIs, you may across a variety of errors. If you want to enable verbose logs to determine what exactly happened, you can set stream logger for Boto in your python script:

# Add this line after importing boto3 library in your python script
boto3.set_stream_logger(name='botocore')

One other thing you may have to deal with, is throttling errors when calling AWS APIs. To configure Boto to retry calls in case of network/throttling failures or to increase connection timeouts:

import boto3
from botocore.client import Config
config = Config(
    connect_timeout=120,
    read_timeout=120,
    retries = {
        'max_attempts': 10,
        'mode': 'standard'
    }
)
session = boto3.Session(
    aws_access_key_id="<dummy access key>",
    aws_secret_access_key="<dummy secret key>",
    region_name="us-west-1"
)
iam_client = session.client(
    'iam',
    config=config
)
print(iam_client.list_user_tags(UserName=<dummy iam username>))

Pagination

One important aspect that is most likely to be missed is the fact that response from Boto library for most of list_* operations are paginated. To ensure you grab all paginated data:

# List IAM all keys
key_list = []
paginator = iam_client.get_paginator('list_access_keys')
for response in paginator.paginate(UserName=USER):
    key_list.extend(response)
# Note that response here is itself a list of subset of keys
print(key_list)

The Boto library is undoubtedly a very powerful tool when it comes to interacting with AWS. Unlike other cloud providers that may require you to download and install multiple different SDKs, Boto is s single comprehensive python package that will allow you to perform any operation on any service that AWS has to offer.