How to use Java High Level REST Client with Spring Boot to talk to AWS Elasticsearch
Recently, for one of my projects, I needed to use Elasticsearch running on AWS Elasticsearch Service domain. When I created the cluster in AWS, I found out that the AWS Elasticsearch domain does not support Transport Client.
Why do I care about Transport Client?
There are two ways to communicate to a remote Elasticsearch cluster:
- Transport Client
- RESTful API with JSON over HTTP
In order to talk to Elaticsearch from a Java application, a lot of people use the Transport Client. For two previous projects, I’ve used Java API and Spring Data. Both of those technologies use Transport Client. I’ve enjoyed Spring Data a lot; it makes your life really easy and that is definitely a reason I care about the TCP. Anyway, long story short, I was left only with option 2. An important thing to note here is that, according to one of their developers, Elasticsearch plans to get rid of the Transport Client in the future. So if you are developing a new application, it might be worth thinking about not using the TCP.
Java High Level REST Client to the rescue!
The great news is that Elasticsearch has developed a library called Java High Level REST Client. So if you are excited about how to use this library then read on.
In this article I will provide step-by-step guidance to build a microservice using Spring Boot which will talk to AWS Elasticsearch Service Domain using JHLRC.
Before I dive into the code, let’s create a use case. Suppose we are developing a small profile application. You can think of it like a child of LinkedIn who is in the embryonic stage. We will be able to create a person’s profile. A profile will have person’s first name, last name, list of technologies, years of experience in those technologies and list of email addresses.
We should be able to:
- Create a profile
- Update a profile
- Find a profile by the Id
- Find all the profiles
- Search all profiles matching a particular technology
- Delete the profile by the id
As simple as that.
Step-by-Step Guide to Use JHLRC
First thing we need is the Elasticsearch cluster in AWS. We will use the free tier which is free up to 750 hours per month and up to 10GB memory.
Please create an account in AWS if you don’t have one; I am assuming you have one. So let’s get started.
Step-1: First create an Elasticsearch cluster. Amazon has a very nice document on how to create an Elasticsearch cluster, also known (by AWS) as an Elasticsearch Service Domain. You can find information on how to create a ES cluster in AWS at the below link.
Getting Started With Amazon Elasticsearch Service
Step-2: If you are reading step-2, I am assuming you have Elasticsearch running in AWS. This means you’ve got the link that AWS has provided you, something like this if you’ve created the domain in the public network:
“search-dev-elasticsearch-5bgsa4vpjib46bgh4cbtebsxydiq.us-east-.es.amazonaws.com”
Once you have received the link above, you can now communicate to Elasticsearch using the HTTP protocol. For example, if you want to see the cluster’s health you can simply type the URL in the browser and the Elasticsearch endpoint like below. Don’t copy this one and paste it in your browser (it won’t work), copy the one you have got from AWS :-)
search-dev-elasticsearch-5bgsa4vpjib46bgh4cbtebsxydiq.us-east-1.es.amazonaws.com/_cluster/health
You should see the cluster information like this:
So now it’s time to create an index in AWS ElasticSearch Service. Use your favorite REST client, or you can use Kibana, which AWS has created for you and given you a link to as well.
Step-3: Let’s send the following mappings and settings to Elasticsearch to create the index called profile.
In the above screenshot, you may have noticed that the data type of the “technologies” field has the type “nested.” If you’re curious why, the reason is that one of our use cases is to be able search profiles by technologies, which are stored in a structure similar to an array. In order to be able to search by an element of an array, the type needs to be “nested”.
Now we have our Elasticsearch cluster ready and we have an index called profile. So Lets go ahead and create our Spring Boot microservice.
Step-4: Head over to Spring Initializer at https://start.spring.io/ and in the Dependencies search field select Web and Lombok. Click “Generate Project”, which will download the project in a zip file.
Step-5: Now lets add the Java High Level Rest Client to the POM file. Open the project in your favorite IDE and add this dependency to your POM file.
Step-6: Now we are ready to write the code. First things first, let’s create a config folder and create a class called ElasticsearchConfiguration. Let’s create an endpoint to create profile information. Here is how it will look:
Step-7 : Let’s add the AWS Elasticsearch URL to the application.properties file.
It looks like this:
Now start the Spring Boot app. If you don’t get any exceptions then we are doing awesome so far :-)
Step-8: Let’s create a Java Bean class which will be mapping to all source fields of the Elasticsearch index. Since Elasticsearch is document-based storage, lets call our Profile Bean class, ProfileDocument. The ProfileDocument = _source in ES.
Step-9: Let’s create a controller so that the client can use our Spring Boot microservice to create profile. Create a class called ProfileController and a class called ProfileService.
Step-10 : You may have noticed that there is an ObjectMapper reference variable in the ProfileService class; we need that to convert our ProfileDocument Object to a Map object. This is needed because when you send the ProfileDocument object to Elasticsearch using the HighLevel Rest Client, you will need to send it either in the form of a Map, or a string in the form of JSON. A string in the form of JSON is not readable and looks messy, which is why I’m using ObjectMapper. In order to use the ObjectMapper, let’s add the dependency to the POM file.
Step-11: So far so good. Let’s create a profile using our microservice. I used Postman to send the profile information.
Step-12: Let’s create an endpoint to find the document we just created. We will create an endpoint in the controller and a method called findById in the ProfileService class.
Add this function to the ProfileController class.
And add the below code to the ProfileService class.
Step-13: Let’s use Postman to find the document.
Step-14: We created the document, now we should be able to update the document as well. Let’s create the update methods. We will create two methods: one called updateProfile in the ProfileController and the other called updateProfile in the ProfileService class.
Now let’s call our updateProfile endpoint using Postman. We will update the first name from “Bob” to “Blah”.
Step-15: Now let’s create the findAll endpoint and the the findAll method in the service class.
The above method returns getSearchResult(searchResponse). I’ve created this method because SearchResponse returns an Array of SearchHit. We need to convert the source response to the ProfileDocument object and make a Java list instead an Array. And I love the Stream class, it’s AWESOME!
Below is how it looks. I’m sure it can be improved upon a lot though :-)
Let’s do some more fun stuff. Now, let’s add an endpoint for searching by technologies. So if there is a profile of a person who knows Angular6, we should get that profile.
Add the following code to the ProfileController.java:
Now let’s create the searchByTechnology method in the ProfileService class. And here we can reuse our friend, the getSearchResult method, that we created before.
Now lets try this out from Postman.
So far so good, right? Let’s do one last thing, which is the Delete operation.
Let’s first create the endpoint.
And the delete method in the ProfileService class.
The code is available at the below link.
References:
