How to access AWS S3 using Boto3 (Python SDK)

In recent times, Amazon Web Services (AWS) has become quite popular in cloud computing. One of the main components is S3, the object storage service of AWS.

We can access S3 through AWS Console, AWS CLI and AWS SDKs of different languages. Boto3 is the name of the Python SDK for AWS. If you’re working with S3 and Python, then you will know how cool the boto3 library is. It makes things much easier to work with.

Prerequisites:

1. Python 3+

2. The boto3 module (pip install boto3 to get it).

3. An AWS account with an AWS IAM user with programmatic access. Add AmazonS3FullAccess policy to that user. This is for simplicity, in prod you must follow the principal of least privileges.

4. Download the access key detail file from AWS console. Create a new file ~/.aws/credential on terminal and paste the details like below:

5. Setup default region in ~/.aws/configfrom terminal.

Basic Operations:

Now that we are ready, let’s start exploring some basic operations.
Creating a Bucket

The first important thing to remember is that the bucket name must be unique throughout the whole AWS platform, as bucket names are DNS compliant. If you try to use a name which another user has already claimed, your code will fail. Instead of success, you will see the following error: botocore.errorfactory.BucketAlreadyExists.

import boto3
s3_resource = boto3.resource('s3')
s3_resource.create_bucket(Bucket="first-aws-bucket-1")

Listing all Buckets in S3

Listing all the buckets present in S3 for that owner.

for bucket in s3_resource.buckets.all():
    print(bucket.name)

Output :

Uploading an Object

Uploading a file Screen_Shot.png from local path /Users/ankhipaul/Documents/Screenshots to S3 bucket created above first-aws-bucket-1

s3_resource.Object('first-aws-bucket-1', 'Screen_Shot.png').\
    upload_file(Filename='/Users/ankhipaul/Documents/Screenshots/Screen_Shot.png')

Output :

Downloading an Object from S3

Downloading an object named doc.pdf from bucket pythonusecase to local system.

s3_resource.Object('pythonusecase', 'doc.pdf').download_file(
    f'/Users/ankhipaul/Documents/doc.pdf')

List all Objects of a specific Bucket

Listing all Objects of the bucket pythonusecase. This will also list all the folders and the files of the respective folders inside this bucket.

pythonusecase = s3_resource.Bucket(name = 'pythonusecase')
for object in pythonusecase.objects.all():
          print(object.key)

Output:

Copy an Object

Copy the object old_convertcsv.csv to a new location within the same bucket with a new name new_convertcsv.csv. Thus, creating a new object.

s3_resource.Object("pythonusecase", "new_convertcsv.csv").copy_from(CopySource="pythonusecase/old_convertcsv.csv")

Output:

Deleting an Object

Deleting the old_convertcsv.csv from bucket pythonusecase

s3_resource.Object("pythonusecase", "old_convertcsv.csv").delete()

Output:

Remember that S3 buckets do NOT have any “move” or “rename” operations. All we can do is create, copy and delete. As there is no move or rename; copy + delete can be used to achieve the same.

Deleting a Bucket

To remove buckets, we have to first make sure that buckets have no objects within them. First, delete all the objects of the bucket first-aws-bucket-1. Then delete the empty bucket.

bucket = s3_resource.Bucket('first-aws-bucket-1')
bucket.objects.all().delete()
s3_resource.Bucket("first-aws-bucket-1").delete()

Output:

Only one bucket is present in S3 after deletion

Encryption of an Object

S3 objects are encrypted using either server-side or client-side encryption. I will be using Server-Side Encryption with Amazon S3-Managed Keys (SSE-S3); in this Amazon S3 encrypts an object using AES-256 algorithm before saving it to disk and decrypts it when you download the objects.

Uploading an image random_pic.jpg into bucket pythonusecase with AES-256 encryption.

s3_resource.Object('pythonusecase', 'random_pic.jpg').\
    upload_file(Filename='/Users/ankhipaul/Documents/random_pic.jpg',ExtraArgs={
                         'ServerSideEncryption': 'AES256'})

On AWS console, we can see the encryption as AES-256.

Versioning

We have all used versioning to protect against accidental deletions or to maintain a complete record of our objects over time. When you request a versioned object, Boto3 will retrieve the latest version.

We have to first enable versioning. In AWS, by default bucket versioning is disabled. Once, we enable versioning, it can be suspended but cannot be disabled.

s3_resource.BucketVersioning("pythonusecase").enable()

Output:

Summary

At this point, we’re done! You should now have a basic understanding of Boto3. For more details, visit https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html

For entire code reference, visit github.