GauravIntegrating Reddit Data with Snowflake Using Docker, Airflow, and AWSThis project showcases an end-to-end pipeline that extracts data from Reddit, processes it using Python, and seamlessly integrates it into…Aug 10Aug 10
GauravTriggering Airflow DAGs with Docker and ngrokAs a data engineer, finding efficient ways to manage and automate workflows is crucial. Recently, I embarked on a project that involved…Jun 16Jun 16
GauravAws Lambda Function: Read file content from s3 bucket on lamda triggerRead file content from s3 on lambda trigger.Apr 16, 2023Apr 16, 2023
GauravApache Kafka - Setup with DockerApache Kafka is a highly scalable and distributed platform for creating and processing streams in real-time.Jan 8, 2023Jan 8, 2023
GauravAws Lambda Function : How we can invoke lambda function inside functionAws Lambda is an event-driven service, which means if any event occur lambda will trigger and we can write any business logic in lambda…Dec 29, 2022Dec 29, 2022
GauravPyspark udf is slow ?A UDF is a basic user-defined function . Instead of pre-defined functions, we can also use UDF(user-defined function ) . UDFs are used to…Aug 28, 2022Aug 28, 2022
GauravAmazon AppFlow — Transfer data from Salesforce to s3Aws AppFlow is a fully managed integration service that lets you securely transfer data between Software-as-a-Services(SaaS) and Aws…Aug 16, 20221Aug 16, 20221
GauravAws Lambda Function -PracticalThe AWS lambda function is a serverless, event-driven compute service that lets your code run manually without managing servers .AWS…Aug 14, 2022Aug 14, 2022
