Invoke Lambda function when we upload data into S3 Bucket — Part -1.

S3 Bucket is the most popular and most used service on the AWS platform.

We already know that we use an S3 bucket to store the data for eg. CSV files, Excel files, Database files and many more.

Whenever we upload and delete any file or perform any operations on S3 Bucket, it will trigger and run the AWS Lambda function.

And I have mentioned the block diagram below which makes you understand what I will do in the whole scenario.

Block Diagram

What we do in this entire scenario

1. We will create an S3 Bucket.
2. We will write a sample code that will print some data like a list of file uploads and content inside the file.
3. After that, we will create an Event.
4. And then we will run the code in AWS Lambda.
5. We will see the output in AWS Cloudwatch.

If you want to create an AWS S3 bucket and put a file via python code you can visit my previous articles AWS S3 with Python — Shoeb Ahmed

In this article, I am going to create an S3 bucket manually and it is very simple and quick.

You can simply log in to AWS Console and visit the S3 Bucket service.

After that click on “Create Bucket”.

Clicking Create Bucket

I have just simply given the name of the bucket and but I am preferring to give the location US East — 1 which is shown below. After that click “Create Bucket”.

Bucket Information

Now I am going to upload the file inside the bucket. And we will write a script to read the file and print the content of the data.

Now I have uploaded the CSV file manually to test out whether my code is working or not for the extraction of data from S3 Bucket.

I am going to write a script that will collect the list of files and print the content of the file, for this scenario I prefer to use a CSV file. That is easy for the beginner level, if you want to extract another file format or download the file in another format, I will help you out and you can ping me on my LinkedIn Account.

After execution of the above code, we can see the output below, can see the list of buckets and the content of the file.

The output of the code

On the right-hand side of that output image, we can see the arrow, which is the bucket we are using and on the left-hand side is the output of the CSV file.

Now we will dive into AWS Lambda.

Before going for AWS Lambda we need to set up our layer that contains required packages for running the code and their functions.

1. First, we are going to create a directory named “packages”.

mkdir packages

2. Now we are going to change the directory and get inside the “packages” directory.

cd packages

3. Now we are going to create a Virtual Environment.

python3 -m venv venv

4. After that we are going to activate the Environment.

source venv/bin/activate

5. Now we are going to create another directory named “python”.

mkdir python
cd python

6. Now I am going to install the required packages.

pip3 install pandas -t .
pip3 install numpy -t .
pip3 install boto3 -t .

7. Now I am going back to the python directory.

cd ..

8. Now we are going to make the zip file of whatever the data contained inside the python directory.

zip -r layer-v1.zip python

I have made a new S3 bucket to store the zip file and that will be used as a layer while running a lambda function.

Notes. Make sure the region of S3 Bucket, Lambda and Layers of Lambda should be the same. I am using the “N.Virginia” region.

I have taken a new S3 Bucket name “layer-for-s3-lambda-v1”.

9. Now I am transferring the zip file into S3 Bucket.

aws s3 cp layer-v1.zip s3://layer-for-s3-lambda-v1/
aws s3 cp zip_file_name.zip s3://s3_bucket_name/

Now we move on to AWS Lambda. And I will show you in Part 2.