Encounter with AWS Lambda and Firehose
Table of Contents
Introduction
Recently I was working on a business requirement to get the data from an upstream service into our AWS Redshift table for analytical purposes. Sounds like a medium sized project right. Well no, actually with AWS and its powerful components I laid a simple architecture and got the data flowing in the redshift table in less than three days.
System Architecture
Architecture Break Down
Our system needed the data from the upstream service. We don’t need the data in real time so asynchronous processing is fine for the use case. Upstream system publishes their data as a SNS notification. I hooked the lambda function onto their SNS notification and got the data into AWS Redshift. Lets see how,
AWS SNS
AWS SNS — Amazon Simple Notification Service (Amazon SNS) is a fast, flexible, fully managed push notification service that lets you send individual messages or to fan-out messages to large numbers of recipients
