What is Elasticsearch Runtime Field And How We Use It?

With Elasticsearch 7.12, the new feature released named runtime fields.
Runtime fields are fields in Elasticsearch that are dynamically generated during query time, instead of being indexed. Runtime fields allow us to perform calculations or making modifications and add additional information to search results without having to index that information beforehand. Runtime fields can be defined at index time or query time. Runtime fields are so flexible that we can add runtime fields without re-indexing and then easily we can delete them.

For example, you can use runtime fields to concatenate multiple fields, calculate a new field value based on existing fields, or format dates to a specific format. These fields can be accessed and used in search queries, just like regular fields in the index.

First of all, lets create our index.

#for creating index
PUT /student_2023_03_15
{
  "settings": {
    "index": {
      "number_of_shards": 1,  
      "number_of_replicas": 0
    }
  }
}

#for adding alias to the index
POST _aliases
{
  "actions": [
    {
      "add": {
        "index": "student_2023_03_15",
        "alias": "student"
      }
    }
  ]
}

Adding the mappings

#adding mapping as usual 
PUT /student_2023_03_15/_mapping
{
  "properties": {
    "id": {
      "type": "long"
    },
    "name": {
      "type": "keyword"
    },
    "lastName": {
      "type": "keyword"
    },
    "age": {
      "type": "long"
    },
    "courses": {
      "properties": {
        "key": {
          "type": "keyword"
        },
        "value": {
          "type": "keyword"
        }
      }
    }
  }
}

#putting some document to the student index
PUT student/_doc/4
{
  "id": 4,
  "name": "Alex",
  "lastName": "Brown",
  "age": 25,
  "courses": [
    {
      "key": "Math",
      "value": "68"
    },
    {
      "key": "Chemistry",
      "value": "56"
    },
    {
      "key": "Physics",
      "value": "59"
    },
    {
      "key": "English",
      "value": "pass"
    }
  ]
}

Now, we have an index named student_2023_03_15 which contains courses field in mapping. This field consists of objects containing key and value. In this courses.value field, we can store numeric or string values as keyword type. Lets say we want to get the average value of all courses. But in this case we have string values such as “failed” or “pass”. With this values, running the avg aggregation of elasticsearch may be difficult. So what do we do? We use runtime fields.

How to Add Runtime Fields?

As we mentioned before, runtime fields can be defined at index time or at query time. Let us start by begin with adding a runtime field to our mapping.

Index Time:

#for all courses that its value is numeric, we add a runtime field

PUT student_2023_03_15/_mapping
{
  "runtime": {
    "chemistry": {
      "type": "long",
      "script": {
        "source": "for(course in params['_source'].courses){if(course.key=='Chemistry'){emit(Long.parseLong(course.value))}}"
      }
    }
  }
}

PUT student_2023_03_15/_mapping
{
  "runtime": {
    "math": {
      "type": "long",
      "script": {
        "source": "for(course in params['_source'].courses){if(course.key=='Math'){emit(Long.parseLong(course.value))}}"
      }
    }
  }
}

PUT student_2023_03_15/_mapping
{
  "runtime": {
    "physics": {
      "type": "long",
      "script": {
        "source": "for(course in params['_source'].courses){if(course.key=='Physics'){emit(Long.parseLong(course.value))}}"
      }
    }
  }
}

After adding these runtime mappings, we can easily perform a range query by using these runtime fields directly.

#searching for the students whose math grades are gte 50 and lte 100
GET student/_search 
{
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "math": {
              "gte": 50,
              "lte": 100
            }
          }
        }
      ]
    }
  }
}

In another way, Query Time:

#by adding the math_long runtime field at query time,
#we can succesfully use the elasticsearch range query on a keyword type 
GET student/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "math_long": {
              "gte": 50,
              "lte": 100
            }
          }
        }
      ]
    }
  },
  "runtime_mappings": {
    "math_long": {
      "script": {
        "source": "for(course in params['_source'].courses){if(course.key=='Math'){emit(Long.parseLong(course.value))}}"
      },
      "type": "long"
    }
  },
  "fields": [
    "math_long"
  ]
}

So how do we use it in Elasticsearch Java API Client :

Script script = Script.of(sc -> sc.inline(is -> is.source("for(course in params['_source'].courses){if(course.key=='math'){emit(Long.parseLong(course.value))}}")));
RuntimeField runtimeField = RuntimeField.of(rt -> rt.type(RuntimeFieldType.Long).script(script));
BoolQuery.Builder boolQueryBuilder = new BoolQuery.Builder();
String fieldName = "math_long";
boolQueryBuilder.must(Query.of(q -> q.range(r -> r.field(fieldName)
                                                  .gte(JsonData.of(50L))
                                                  .lte(JsonData.of(100L)))));
SearchRequest searchRequest=  SearchRequest.of(sr -> sr.index("student")
                                .runtimeMappings(fieldName, List.of(runtimeField))
                                .query(q -> q.bool(boolQueryBuilder.build())))
try {
    esClient.search(searchRequest, Object.class);
} catch (IOException e) {
    e.printStackTrace();
}

Advantages and Disadvantages of Runtime Fields

Advantages:

• Basically, the main advantage of runtime fields is that they can later add new fields to created documents.
• Runtime fields can provide dynamic information, such as the current date or time, without having to update the index.
• Because runtime fields aren’t indexed, adding a runtime field doesn’t increase the index size.
• When you define a runtime field, you can use it immediately in search requests, aggregations, filtering, and sorting.
• If we want to change a runtime field into an indexed field, you don’t need to modify any queries that refer to the runtime field.

Disadvantages:

• Runtime fields require less memory and provide more flexibility in accessing your data, but may impact search performance based on the computation defined in the runtime script or large data sets.
• Using runtime fields can increase the complexity of your search and aggregation queries, making them harder to debug and maintain.

In summary, Elasticsearch runtime fields offer a flexible and efficient way to derive new fields on the fly, without having to re-index your data. By leveraging runtime fields, you can easily add new data transformations, perform complex computations, and create custom aggregations on your data in real-time. With this powerful feature, you can unlock new insights and make more informed decisions based on your data. So, if you’re looking for a way to improve the performance and flexibility of your Elasticsearch queries, consider adding runtime fields to your toolkit.

Thank you for taking the time to read this article on Elasticsearch Runtime Field. I hope you found it informative and helpful. If you have any questions or feedback, please feel free to leave a comment below.