S3 Basic File Operations Python Cheatsheet

Introduction

The easiest way to download, upload, copy or delete files and folders from S3 buckets is using the AWS Management Console. However, there are cases where you have to use a dedicated SDK tool, such as:

• Bulk file operations are needed
• Your organization limits the usage of AWS console in order to increase security

In these cases, you have to perform the mentioned operations using the AWS CLI tool or SDKs. As a Python developer I’m using the SDK for Python named boto3.

Boto — Amazon river dolphin

However, when I searched for code examples for deleting a file from S3 bucket with Python, or how to copy and delete a folder in S3 bucket using boto3, I received many answers and it was hard to reach the right one that met my case. In addition, those operations with S3 buckets were scattered around many different resources. Therefore, I decided to keep them in one dedicated document which I am happy to share it here with you.

What is boto3?

Boto3 is maintained and published by AWS.
According to PyPI documentation of boto3:

Boto3 is the Amazon Web Services (AWS) Software Development Kit (SDK) for Python, which allows Python developers to write software that makes use of services like Amazon S3 and Amazon EC2.

In our case, boto3 allows us to perform operations like download, upload, copy or delete files and folders on S3 buckets using Python. It does much more than that, and you can learn more about it in Boto3 documentation.

Fun fact

Boto (pronounced boh-toh) was named after the fresh water dolphin native to the Amazon river. The name was chosen by the author of the original Boto library, Mitch Garnaat, as a reference to the company.

Prerequisite

Before we dive into the code, make sure you have installed boto3 and set up the required credentials. For more information, check PyPI’s install guide for boto3.

Upload a file to S3

You can upload any type of file from your local path into the path in your bucket. You can change the file name that is being saved in the file_destination_key. If the file should be located in a specific sub folder, give the full path of those sub folders in file_destination_key.

# Upload file to S3
import boto3
client = boto3.client('s3')

bucket = "my_bucket_name"
file_local_path = "path/to/my_file.ext"
file_destination_key = "path/of_the_file_in_the_bucket/new_file_name.ext"
client.upload_file(file_local_path, bucket, file_destination_key)

Also, it is recommended to grant specific access control lists (ACL) to the file you upload. For example, this will grant the bucket owner full control of the new file:

# If you want to add specific access control lists (ACL) to the files, you can add it this way
# For example, grant the bucket owner full control of the new file
boto3.resource('s3').ObjectAcl(bucket, file_destination_key).put(ACL='bucket-owner-full-control');

Download a file from S3

# Download a file from S3
import boto3
s3 = boto3.resource('s3')

bucket = "my_bucket_name"
file_key_name = "path/of_the_file_in_the_bucket/to_download.ext"
file_local_path = "path/to_the_local/downloaded_file_name.ext"
s3.Bucket(bucket).download_file(file_key_name, file_local_path)

Delete a file from S3

File delete operation is a dangerous command to execute so please be sure the file path you enter is the correct one :)

# Delete a file from S3 - please make sure first that it's the right key!
import boto3
client = boto3.client('s3')

bucket = "my_bucket_name"
file_key_name = "path/of_the_file_in_the_bucket/to_delete.ext"
client.delete_object(Bucket=bucket, Key=file_key_name)

Copy folder into another folder in S3

Copy folder into another folder in S3 will work as long as the folders (or buckets) are located in the same region. If the new folder is located in another region, you will have to download each file locally and only then upload it to the new bucket. If you are familiar with a better solution, please let us know by leaving a comment below.

# Copy folder into another folder in the same region
import boto3
client = boto3.client('s3')

bucket = "my_bucket_name"
new_bucket = "new_becket_name"
old_key_folder_name = "path_to_old/folder/"
destination_key = "path_to_new/folder/"
response = client.list_objects_v2(Bucket=bucket, Prefix=old_key_folder_name)
for key_loc in response["Contents"]:
    key = key_loc["Key"]
    key_to_copy = {'Bucket': bucket, 'Key': key}
    client.copy_object(Bucket=new_bucket, CopySource=key_to_copy, Key=destination_key)

Delete a folder in S3

There are cases where you no longer want to keep a folder in an S3 bucket. For that, you want to delete all its content — files and files in subfolders. Please note that list_objects_v2() will return some or all (up to 1,000) of the objects in a key path with each request.

Same delete warning here as well — please validate that the folder path you want to delete in S3 is accurate.

# Delete a folder from S3 - please make sure first that it's the right key!
import boto3
client = boto3.client('s3')

bucket = "my_bucket_name"
old_key_name = "folder/to_delete/"
response = client.list_objects_v2(Bucket= bucket, Prefix=old_key_name)
for key_loc in response["Contents"]:
    key = key_loc["Key"]
    client.delete_object(Bucket=bucket, Key=key)

Bonus

It is also possible to read or write files like CSV or JSON to S3 buckets using the pandas package. However, boto3 handles all the file types.

In summary

In this blog we covered download, upload, copy and delete files and folders from S3 buckets using boto3. This is my go to doc when I need to perform these operations in my daily work. I hope that you find it as useful as I did, and that it will save you valuable time.