Elastic Search Query Framework

Context

We have leveraged ElasticSearch for delivering a search solution on our platform and it is performing seamlessly over last couple months.

Some of the challenges we needed to address in the design were around search queries. Mainly,

• Development turn-around time for any change was more than a day
• Query change meant a lot of code change
• Index/alias name change required code changes
• Query versioning
• Code re-usability

To address these challenges, we leveraged the query framework. The proposed solution looks like below.

Re-factored the search solution by developing below odules:

1. Search Configurator
2. Search Templates
3. Search Query Template Store
4. Search Domain Adapter

Search Configurator

This module allows the users to configure the indices, which are used for searching the desired dataset from the search engine. This is completely configurable, and users can define/ configure,

• The index used for querying
• The filters to be applied on the index
• The sorting options at field level of the index
• The Routes each index supports
• The type of searches (full text or auto suggestion) the index supports
• The hooks to implement custom data transformation

Objective of this module is to extend the search capabilities with configuration when we introduce new indexes rather than changing the code.

Search Templates

The templates are used to define the queries to be executed. The templates are specific to each search engine. One should have the knowledge of search engine specific query language to enhance these templates. The framework leverages Search Configurator and the Search Templates to prepare request body/ query parameters.

Template Store

This is a repository to store Search Templates. At the moment, we are using AWS S3 as the template store. In future, we look to use AWS Dynamo (or any NoSQL database) as the template store.

Search Domain Adapter
This component is designed based on adapter pattern. It’s functionality is to parses the templates and generates DSL. Once the DSL queries are generated, elastic client which is embedded as a library within the adapter runs these queries.

Due to this design,

1. Any changes like adding new index or configuring the fields of index to retrieve data can be achieved in a day without writing a single line of code
2. Custom data transformation rules can be defined in the configuration file for data enrichment before executing the query. This could be achieved without writing a single line of code

Overall, our ability to deliver changes and new functionalities has increased reducing the time to change to hours from days, significantly bringing new business agility.