AWS Lambda to Search and Read Data using ElasticSearch

Fig.1: basic workflow between AWS Lambda and Amazon ElasticSearch Service

In this blog, I will give a walkthrough on how to use AWS Lambda to perform various tasks in ElasticSearch.

This blog assumes that you are having a basic knowledge of AWS Lambda, ElasticSearch service, and Python.

AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume.
Amazon Elasticsearch Service is a fully managed service that makes it easy for you to deploy, secure, and run Elasticsearch cost-effectively at scale.

According to Aamazon Web Services

In this blog, I’m going to explain the following steps which will help you to write a python Lambda for using ElasticSearch service.
1. Installing Required Packages.
2. Configuring your lambda function to connect ElasticSearch service.
3. Writing ElasticSearch queries and sending the request.
4. Reading the ElasticSearch response or result.
5. Uploading your code with the required packages to AWS Lambda.

1. Installing Required Packages python for AWS Lambda:

I’m using following external packages for handling ElasticSearch from lambda:
1. requests_aws4auth:
Use the below command to install requests-aws4auth through pip.
pip install requests-aws4auth
requests-aws4auth is tested on Python 2.7 and 3.3 and up.

2. requests:
Use the below command to install requests through pip.
pip install requests

note: The above commands will install requests and requests_aws4auth packages in your local system. This will be useful when you're accessing the ElasticSearch service from your local code.

Installing packages in a targeted path:
Use the below commands with specific path after --target to install under the specific file path.
pip install --target ./toPath requests
pip install --target ./toPath requests-aws4auth
Take the reference to the below snapshot.

Fig.2: Installing packages using pip

In the above snapshot, we are installing requests-aws4auth under D:/packages/. It will be helpful while you're uploading your code to AWS Lambda (5).

2. Configuring your AWS Lambda function to connect ElasticSearch service:

I’m assuming that you’re already having an ElasticSearch domain. If you’re not created domain then click here for more information.
1. import the packages from your lambda:
Paste the following code on top of your Lambda function
import json
import requests
from requests_aws4auth import AWS4Auth

2. Getting an ElasticSearch endpoint:
go to your AWS account-> ElasticSearch Service-> domain-> endpoint
Let's take look on the below image, which will help you to get the ElasticSearch endpoint.

Fig.3: shows how to get an ElasticSearch endpoint.

Kibana is the test platform to test your es-queries before adding a query to your code. If you want to test your es-queries then click on Kibana-URL and you will be redirected to the Kibana dashboard after authentication.

Fig.4: Kibana Dashboard

Click on console in your Kibana dashboard. The console window will be opened and you can test your queries as shown below:

fig.5: Kibana console with the example search query

Let’s take a look at the above image. It contains 2 windows. The left window is your search query and the right window contains the result for your query.

3. Setting Up your lambda function to call ElasticSearch service:
Take a reference of the below code to configure ES in your lambda.

3. Writing AWS ElasticSearch queries and sending the request:

Writing queries:
In ElasticSearch service you can write different types of queries based on your requirement. In this blog, I will demonstrate some of the basic and most important queries.
1. match_all: To retrieve all the rows without any condition use the following query.

2. match: If you want to retrieve the data which matches some condition then use match query. The below query is used to get the rows that match the given condition.

The above query returns the maximum 160 rows which are having field name as test
note: size is used to mention the maximum number of rows you want to get in the result. The maximum range for size is 10000.

3. compound queries: Compound queries are used to combine match query with filter range. filter is used to retrieve the rows which have a value between the given range. The below code is the example of the compound query.

note: The above query returns the rows where course field match to BCA and joinedDate ranges from the last 30 days.
To know more about compound queries click here.

4. multi_match queries:
To match the same value with multiple-fields(more than one field) multi_match will be used. The below example matches the keyword "surprise"and "test" with 2 different fields "subject" and "message".

same as

To know more about multi_match click here.
Note: Above, I’d demonstrated a few important and most commonly used ElasticSearch queries. To know more about queries visit the ElasticSearch site.

Sending query Request to ElasticSearch:
The below code is an example for calling ElasticSearch service from your lambda function through request package.

4. Reading the ElasticSearch response or result data:

The result from ElasticSearch will be decoded from JSON format and will be saved in the result variable. To work with the result take a reference of the following code.

The above code is an example for reading data from the result. Before accessing particular field value from the result don’t forget to test your query in Kibana console and make sure that your accessing field matches the Kibana result field.

5. Uploading your code in AWS Lambda:

If you wish to upload your function to “AWS Lambda” then follow the below steps:

• Archive(.zip) python file(es.py) along with the packages (files: requests,requests-aws4auth).[ex: test.zip]
• go to -> AWS dashboard -> Lambda -> create function -> enter FunctionName -> runtime(Python 3.8) -> create function.

fig.6: Creating a function in AWS Lambda

• Once function created goto -> code entry type(under function code) -> upload a .zip file->upload(under Function Package) -> upload your test.zip file.
• In Handler info -> update your filename_without_extension.main_function_name(ex: es.es_search_main, don't add .py to the filename).

Fig.7: Uploading .zip file in AWS Lambda

Conclusion

I hope you liked and followed this blog using ElasticSearch service in Python AWS Lambda. In this blog, we covered installing packages, getting an endpoint, setting up lambda function with endpoint and queries, handling the ElasticSearch result in lambda function and uploading the code with the required packages to AWS Lambda. If you’re having any queries please free to comment below.

For more blogs and research visit Tensult blog

References:

Elastic Stack and Product Documentation

Originally published at https://blogs.tensult.com on January 1, 2020.