Loading Data to AWS Elasticsearch with DynamoDB Streams and Lambda

Elasticsearch is an open-source, RESTful, distributed search, and analytics solution that is currently widely used by many companies worldwide. It is a search engine based on Apache Lucene library. Elasticsearch is widely used in text analysis, log analysis, text-based search and many more use cases.

Amazon Elasticsearch is a fully managed service that makes it easy for us to deploy, secure, and run Elasticsearch. The service provides support for open-source Elasticsearch APIs, Kibana integration with Logstach, and other AWS services. In this post I am going to talk about how we can load streaming data into AWS Elasticsearch service using AWD DynamoDB streams and AWS Lambda. So in our scenario when a web application or a web service adds data to our dynamoDB database, it will trigger an AWS Lambda function which will automatically insert those data to our Elasticsearch service which we can use in our use case.

I will assume for this post you have a basic idea on what is Elasticsearch, dynamoDB, and Lambda. Hence in this post I am not going to discuss thoroughly what Elasticsearch, dynamoDB, and Lambda does. Ok, enough on explanations…