How to Send a CSV File from S3 into Redshift with an AWS Lambda Function

Introduction

Nowadays is a must to automate everything and cloud jobs are not the exceptions, as Data Engineer We need to acquire the skill of move data wherever needed, if we want to know how to start facing AWS tools in your daily routine like a data professional, this post is for you.

Step By Step

After collecting data, the next step is to design an ETL in order to extract, transform, and load your data before you want to move it into an analytics platform like Amazon Redshift in this case, only We're going to move data from S3 into a Redshift Cluster using for AWS free tier.

To do that, I’ve tried to approach the study case as follows :

1. Create an S3 bucket.
2. Create a Redshift cluster.
3. Connect to Redshift from DBeaver or whatever you want.
4. Create a table in your database.
5. Create a virtual environment in Python with dependencies needed.
6. Create your Lambda Function.
7. Someone uploads data to S3.
8. Query your data.

¡¡Let’s get started !!