Extract zip file in S3 Bucket using AWS lambda functions in nodeJs
Github Link for the code : https://github.com/stirpude/s3-lambda-nodeJS-zip-file-extract
Hello Everyone,
I had been struggling with uploading zip folder and unzipping the contents in AWS S3. AWS S3 does not provide the functionality of unzipping the zip folder in the S3 bucket directly.I had been searching the solution online, but did not get solution using NodeJS. The documentation provided by AWS is as always confusing and you are generally lost after reading it and even might give up attempting the problem statement altogether.
I got few solution using AWS lambda functions and Python, but I wanted to use NodeJs as I am familiar with it.After working on it a couple of days, I finally found a solution to the problem statement of unzipping the compressed file in AWS S3 using AWS lambda function and nodeJs.
Here is the overview of the processes that are carried out :
1.Upload a zip file to the S3 bucket
2.The upload triggers the lambda function to extract the zip file and store it in the same bucket.
Lets look at the steps one by one to extract the zip file in S3 bucket using nodeJS.
Step 1 : Create a lambda function
Create a lambda function in AWS.Upload a zip folder containing the following index.js file.We are using S3-unzip library to unzip an archive file in a S3 bucket to its root folder.Full code can be accessed here.
Install S3-unzip :
- To include as library:
npm install s3-unzip
You can find the index.js file here.
Step 2 : Attach IAM policy
Create an IAM policy giving the following permissions.At first, I applied the AmazonS3FullAccess to the lambda function, but it is not advisable to give full access in production.Hence, following IAM policy can be applied to the lambda function in production environment :
Step 3: Add a trigger to lambda function
Whenever a zip file is uploaded to the S3 bucket, lambda function should be triggered to extract the zip file in same bucket.We need to add a trigger to lambda function.Click on Add trigger and select S3 from the list.Provide the s3 bucket name and select on Suffix as .zip. Once added, this will be triggered whenever there is a zip file upload on S3 bucket.
Step 4: Upload the module dependencies in the Layers of lambda function.
We cannot upload the node modules dependencies required directly to lambda functions because the zip file might exceed the size 10 MB.Hence we need to put the node modules folder in the Layers.Create a layer providing name and upload a zip file named nodeJs containing the node module folder dependencies required by lambda function. You refer the below article for creating Layers in AWS Lambda
Layers Article
https://medium.com/@stirpude/layers-in-lambda-function-for-node-js-988c07268565
This way, whenever there is a zip upload on S3 bucket, lambda function will be triggered to extract the zip file in the same folder.
