Running up Elasticsearch on Amazon Web Service (AWS) with NodeJS
Elasticsearch what and why ?
Based on my reading and discovery from some videos, Elasticsearch is a distributed restful search and analytical engine which built on top of Apache Lucene. It’s easy to use, scalable, support full-text search and use document based index rather than table or schema. So it has no transaction, like any other database, no relations, no constrains, and it also schema free.
It has a lot of query mechanisms, because it built for search purpose. It based on restful and JSON mapping, easy to integrate with any programming language, you even could perform direct search by using a query parameters through Elasticsearch service API. It absolutely fast everywhere, both searching and indexing. Which also support node clustering, document shards and replication. For more understanding please read through this official documentation of Elasticsearch.
How Amazon Web Service could help ?
Common problem occurs when talking about building a tech architecture is about to maintain any services.
It is easy to build and run some services although hard to maintain and keep it stable for long term usage.
Like any other services, Elasticsearch need an instance to be used. Moreover you work with a company with a lot of data to be indexed and able to be searched as fast as possible to serve a good user experience.
Amazon web service comes to solve maintenance problem by helping to manage the Elasticsearch services cluster, so no more hassle to handle its deployment, auto scaling, monitoring and security. The clusters endpoint created within AWS Elasticsearch could simply opened as public or secured by privatising it through AWS virtual private cloud (VPC).
Setting up AWS Elasticsearch cluster
Lets now move to setting up an Elasticsearch instance in AWS using Amazon Elasticsearch Service. Open your Amazon Elasticsearch Service dashboard and click on create new domain.
You will see a step by step to instantiate a new Elasticsearch cluster. Choose an instance type for development and testing. And at the time I write this tutorial the latest version of Elasticsearch itself is version 6.7 but AWS currently support up to version 6.4 only.
Now choose the name of your Elasticsearch cluster as shown above, I named it mgf-es as the name and choose t2.small.elasticsearch as the smallest instance for Elasticsearch available and put 1 number of instance also. And just ignore remaining options as default, then proceed to next step.
This step determine access to your cluster endpoint. At this point we follow the most recommended access which is VPC access. Define your VPC, subnet and security group. If you want another instance to be able to connect to your Elasticsearch endpoint you should use same security group.
After that select a domain access policy, if we already use VPC its okay to choose first option which `Do not require signing request with IAM credential` because the Elasticsearch endpoint already secured with VPC and cannot be accessed publicly.
Now you have a running Elasticsearch cluster within secured VPC endpoint as seen above, also AWS provide a secure access to Kibana (GUI dashboard for Elasticsearch), but it will only be accessible using AWS Cognito and I will not discuss this now.
There we are, you have already a cluster of Elasticsearch which able to scaled manually in easy manner from its configuration. Next we try to run a Nodejs application from Elastic Beanstalk and try to connect this Elasticsearch VPC endpoint
Build search API with nodejs
I have made some Nodejs application code which consists of connection to Elasticsearch itself, seeding the data by indexing a dummy data, and search endpoint so we able to search seeded data which indexed while initiate the application.
- Go to https://github.com/ghiyastfarisi/node-es-playground and clone the repository.
- Create .env file from sample.env file and define your Elasticsearch VPC endpoint there.
Next we build an instance in AWS Elastic Beanstalk to deploy the application in simple way. Go to Elastic Beanstalk dashboard and choose get started.
Continue to configure more options, then choose to modify the configuration in instances tab. We need to checklist the same group id which also used in Elasticsearch VPC instance. As shown below, I use sg-71876417 as group id.
Then move to modify the configuration in network tab, and use same VPC group as the Elasticsearch instance we created before. As can be seen below vpc-6f5f0d0b is used for this instance and Elasticsearch instance before.
Last configuration which located in software tab, is to define the environment variable used for this instance. As we can see in .env or sample.env in the application, we need 3 configuration which are NODE_ENV, ELASTICSEARCH_URI and PORT.
Finally after modifying those all configurations, we ready to launch the instance by clicking Create Application . It will take several minutes to be settled up.
Be ready, if the application already up and running, you can now deploy the nodejs application which cloned from github before by zipping all it in .zip format and choose Upload and Deploy.
After deploy executed, wait several minutes for the Elastic Beanstalk to upload and deploy your application. No need other config to run, because default nodejs application in Elastic Beanstalk will use package.json standard command to run nodejs application which is npm start as the main, and the EB also automatically run npm install as the default command before running the nodejs application.
After the deployment finished, open the url of Elastic Beanstalk endpoint and add /search after the url to go to search API created in application. You will see like in the picture below, our pokemon index in nodejs application running as well.
Thats it, you have finished deploying a cluster of Elasticsearch with simple nodejs application to handle indexing and searching a document. For more understanding how the indexing and searching works, you can learn by reading the code from the application or just read the official documentation from Elasticsearch. Hopefully this will help you guys, learning such an amazing Elasticsearch as a tools for search engine. Thanks for reading this writing.
See ya.
