Boto3: AWS SDK for Python

Introduction :

By now, it’s well-known that AWS(Amazon Web Services) is the biggest cloud computing platform. Even though Microsoft Azure and other platforms are catching up slowly, AWS will not be dethroned anytime soon. AWS can be controlled using many ways like through the AWS Console, AWS CLI, Python, Java, and other programming languages.

Boto3 is the AWS SDK(Software Development Kit) for Python to create, configure, and manage AWS services, such as Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3). The SDK provides an object-oriented API as well as low-level access to AWS services. Boto3 is maintained and published by AWS. Boto (pronounced boh-toh) was named after the freshwater dolphin native to the Amazon river. The name was chosen by the author of the original Boto library, Mitch Garnaat, as a reference to the company.

We can find the latest documentation at the Boto3 doc site, including a list of services that Boto3 supports, and its source code here

Boto3 Usage :

The Boto3 library allows us to easily integrate our Python applications, libraries, or script with AWS services and gives us full control over all AWS services like Amazon EC2, Amazon VPC, Amazon S3, Amazon DynamoDB, and others. Although, we need to keep in mind that the user or service making the API calls must have the actual permissions to call AWS APIs. Those permissions are controlled by the AWS IAM service, defined in IAM Policy, and granted to the user or service through the IAM Role.

To start using the Boto3 library to interact with AWS APIs, we have to install the Python boto3 module, import it from the Python program code, and use boto3.client() or boto3.resource() method to connect to a specific AWS service API endpoint. Finally, we would have to use the Boto3 client or resource object methods to interact with service-specific API.

Installation :

If your computer does not already have Boto3 or AWS tools, you should install them first. There are two Python modules we need to be able to manage AWS services from the command-line (CLI) and your Python scripts:

• boto3 — the AWS SDK for Python
• awscli — command-line tools allowing you to manage AWS services from your terminal

We need to install the AWS CLI tools to configure the AWS environment on your machine.

To install Boto3 on your computer, go to your terminal and run the following command:

pip install boto3

We have the SDK, but we won’t be able to use it right away. This is because the SDK doesn’t know which AWS account it should connect to.

To make it run your AWS account, you’ll need to provide valid AWS credentials. If you already have an IAM user with full permissions to S3, we can use those users’ credentials (their access key and their secret access key) without needing to create a new user. Otherwise, create a new AWS user and then store the new credentials.

To install the AWS CLI tools, you have to run another command in your terminal:

pip install awscli

Configuring AWS environment :

AWS CLI is a set of command-line tools for accessing AWS from the terminal shell. Those tools are available for us through the `aws` command. We will use the subcommand `configure` to set up an AWS environment on your workstation. To configure the AWS environment, type the following command in your terminal:

aws configure

This command will walk you through an environment configuration process and ask you for 4 things:

• AWS Access Key
• AWS Secret Access Key
• Default region name –> enter us-east-1
• Default output format –> enter json

The `aws configure` command updates a text file that the awscli, boto3, or any other software that interacts with AWS. If you are on macOS or Linux user, the aws configure command updates the text file at this path:

~/.aws/credentials

**NOTE: The aws configure tool allows you not to store your AWS credentials (the AWS Access and Secret Keys) in your Python scripts.

Passing AWS credentials to Boto3 Client Directly :

In cases where we cannot use an external tool like AWS CLI, we can directly use them in the client() as parameters. To pass AWS credentials to the Boto3 client, you have to provide them in the aws_access_key_id and aws_secret_access_key variables, region_name variable for example:

import boto3
client = boto3.client(
‘ec2’,
aws_access_key_id=”XXXXXXX”,
aws_secret_access_key=”YYYYYYY”,
region_name=”us-east-1"
)

**NOTE: Storing your AWS credentials in your scripts is not secure and, you should never do this, we can set them as environment variables or use `.env` file and load it into the Python script but even storing AWS Access and Secret Keys in a plain text file is not very secure. The better and more secure way is to store AWS Access and Secret Keys in the encrypted store, for example, aws-vault.

Testing AWS credentials via AWS CLI :

A common way of testing AWS credentials is by running the following command:

aws sts get-caller-identity | tee

This command will produce information about your AWS Account and User IDs. The output will look like this:

{
“UserId”: “AKIAYQV4J5VM5FWGJMGB”,
“Account”: “012345678901”,
“Arn”: “arn:aws:iam::012345678901:user/admin”
}

Testing AWS credentials via Boto3 :

We can request the same information by using the Boto3 library. Here’s an example of a Python script to do that:

#!/usr/bin/env python3
import json
import boto3
client = boto3.client(‘sts’)
response = client.get_caller_identity()
user_id = response[‘UserId’]
account = response[‘Account’]
arn = response[‘Arn’]
output = {
`‘UserId’: user_id,
‘Account’: account,
‘Arn’: arn
}
print(json.dumps(output, indent=4))

In the Python code above, we did :

• Import of a built-in json module to process JSON response from theget_caller_identity() method — (line 3)
• Import of the boto3 module to have an ability to make calls to AWS APIs — (line 4)
• Instantiate the AWS Security Token Service (STS) client — (line 6)
• Make a call to the AWS STS service to get information about AWS Account and User IDs and save the service response to the variable — (line 8)

**NOTE: All AWS API calls at the Boto3 library return response information in the form of a Python dictionary.

Now, run the Python script to get the same output as you got from the AWS CLI call

Botocore :

The AWS CLI and Boto3 library are built on top of a standard botocore module. This module is a low-level Python library that takes care of low-level operations required to send secure HTTPS API requests to AWS and respond. The botocore module is responsible for:

• Configuring and handling AWS user sessions and credentials during API calls
• Providing a fine-granular interface to manage low-level AWS API calls to a specific service (for example, AWS STS GetCallerIdentity API call)
• Serializing and deserializing input parameters, and XML responses received from API calls into Python dictionaries
• Providing low-level clients and high-level resources abstractions that we’re using to interact with AWS services from Python

In short, botocore is a utility module that allows you to forget about dealing with really low-level API calls to AWS services and use more high-level methods to interact with them.

Client Vs Resource :

At its core, all that Boto3 does is call AWS APIs on your behalf. For the majority of the AWS services, Boto3 offers two distinct ways of accessing these abstracted APIs:

• Client: low-level service access
• Resource: higher-level object-oriented service access

You can use either to interact with S3. We started using the Boto3 client to make an API call to the AWS STS service. At the same time, you may find lots of examples on the internet, where people are using the client() instead of the resource() method. So, what are the differences?

The major differences between boto3.client() and boto3.resource():

In summary, using the resource() allows us to write a more readable code and avoid processing response dictionaries. In addition to that, it allows you to lazily iterate over a large number of returned objects without thinking about pagination or memory utilization.

Listing IAM Roles using Client or Resource :

As an example for comparing both APIs, let’s get all IAM Roles from the AWS account using the client():

import boto3
client = boto3.client(‘iam’)
paginator = client.get_paginator(‘list_roles’)
aws_roles = []
for page in paginator.paginate():
 for role in page[‘Roles’]:
  aws_roles.append(role[‘RoleName’])
print(‘\n’.join(aws_roles))

Now, let’s do the same thing but with the help of resource() instead:

import boto3
iam_resource = boto3.resource(‘iam’)
aws_roles = []
for role in iam_resource.roles.all():
 aws_roles.append(role.name)
print(‘\n’.join(aws_roles))

As you can see, the resource() simplifies our code by removing the need to use a paginator and additional nested for-loop.

We can also get access to the client() methods from the resource():

iam_resource = boto3.resource(‘iam’)
# getting access to the client from resource object
iam_client = iam_resource.meta.client
paginator = iam_client.get_paginator(‘list_roles’)
aws_roles = []
for page in paginator.paginate():
 for role in page[‘Roles’]:
  aws_roles.append(role[‘RoleName’])
print(‘\n’.join(aws_roles))

Sessions in Boto3 :

The boto3.Session() is a class that stores configuration state and allows you to create the Boto3 clients() and resources(). Dealing with multi-region and multi-account AWS API calls topic is out of the scope of this article. Still, I have to mention that it is preferable to use session when you’re working with multi-region or multi-account code:

import boto3
session = boto3.Session()
s3 = session.client(‘s3’)
ddb = session.resource(‘dynamodb’)

Using boto3.Session() becomes even more important when you need to assume roles across different AWS accounts.

Assuming IAM role using Boto3 :

To assume an IAM role using Boto3 you need to use the assume_role() method of the STS client.

**NOTE: you need to make sure, that your current user has been granted permission to assume that role.

import boto3
CURRENT_ACCOUNT_SESSION = boto3.Session()
STS_CLIENT = CURRENT_ACCOUNT_SESSION.client(‘sts’)
AWS_ACCOUNT_ID = ‘585584209241’
ROLE_NAME = ‘demo-role’
assumed_role_object = STS_CLIENT.assume_role(
RoleArn=f’arn:aws:iam::{AWS_ACCOUNT_ID}:role/{ROLE_NAME}’,
RoleSessionName=f’{ROLE_NAME}-Session’
)
assumed_role_credentials = assumed_role_object[‘Credentials’]
ASSUMED_ROLE_SESSION = boto3.Session(
aws_access_key_id=assumed_role_credentials[‘AccessKeyId’],
aws_secret_access_key=assumed_role_credentials[‘SecretAccessKey’],
aws_session_token=assumed_role_credentials[‘SessionToken’],
)
instances = ASSUMED_ROLE_SESSION.resource(‘ec2’, region_name=”us-east-1").instances.all()
for instance in instances:
print(f’Instance ID: {instance.id}’)

Conclusion :

At this point, we’re done! It would be best if you now had a basic understanding of Boto3 with the examples given here. May this tutorial be a stepping stone in your journey to building something great using AWS!