Interact with AWS S3 using Python 3

Anirban Som
Nov 4 · 4 min read

AWS S3 or Simple Storage Service is a service offered by AWS that provides object storage through web interface.

Below is the architecture how Python interacts with AWS S3 –

Architecture

AWS SDK for Python which is boto3 interacts with AWS S3. It can read data from S3 and can write data into S3.

There are many ways through which we can interact with S3. In this article we will discuss how we can interact with AWS S3 using python. But before that let us understand some of the fundamental concepts of AWS S3.

AWS S3 works with a concept called Buckets & Objects. Bucket you can think of as a container and Objects is file that you store inside the Bucket.

AWS S3 Architecture

Steps to create Bucket in AWS S3 –

1. Go to AWS Management console.

2.Open Simple Storage Service (S3)

3. Click on Create Bucket

You need to select Region while creating a Bucket. Your Bucket name should be globally unique in the selected Region.

4. Once your bucket is created you are ready to store files (Objects) into the bucket. You can also create folders inside a bucket to arrange your data.

We will perform below activities with S3 using Python –

1. List down all the buckets present in S3

2. Upload a file from local to S3 bucket

3. Download a file from S3

4. Copy a file from one Bucket to another.

Prerequisites

1. Install Python 3.x

2. Install aws package (pip install aws) to configure Access Key and Secret Access Key

3.Install AWS SDK for Python (pip install boto3)

4. Install Python IDE (PyCharm)

List down all Buckets present in S3

Code

In this case I have 2 buckets in my S3. Let’s see the response dictionary returned by AWS –

AWS Response Object

And, in the final output you will get a list of all the S3 buckets –

Output

Upload a file from local to S3 bucket

Code

In this script we are copying sales_data_201903.csv’ file from our local to S3 bucket

Bucket before running the script –

Bucket after running the script –

Download file into local driver from S3

We will download the file ‘sales_data_201903.csv’ from anirban-som-main-bucket bucket into our local drive.

Folder before running script –

Folder after running the script –

Copy file from one Bucket to another Bucket

Here we are copying the file ‘sales_data_201903.csv’ which is present in anirban-som-main-bucket Bucket to anirban-som-sub-main-bucket Bucket.

Before running –

After running the script –

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade