Mappings in Elasticsearch — A short introduction

Phase 02 — indexing, mapping and analysis — Blog 07

Introduction

In this short blog, I will explain what is mapping in elasticsearch along with some common useful best practices. A good understanding of mapping will be handy, when we learn analysing/analyzers in Elasticsearch later in this blog series.

Mapping

Mapping is nothing but the schema for the documents in Elasticsearch. As I have mentioned before in one of the blogs in Phase 01, Elasticsearch is schema less. This means, unlike other databases like MongoDb or MySql, there is no need to pre-define the schema for the document before we are indexing it to Elasticsearch. So how does this work?. It is simply that Elasticsearch has the built in ability to detect the type of the fields of a document and generate a schema and apply it throughout the index. Of Course there are some things to be taken care of here, but we will come back to it later in this blog. First let us see how the mapping comes into existence when we index a sample document.

Step 1

Create an index without any documents.

curl -XPUT localhost:9200/testindex-0202

We will get the following response for the above command:

{
  “acknowledged”: true,
  ”shards_acknowledged”: true,
  ”index”: ”testindex-0202"
}

Now let us see if there are any mapping applied to it by making use of the “get mapping” API like below:

curl -XGET localhost:9200/testindex-0202/_mapping

The above command will result in the following response:

{
  “testindex-0202”: {
    “mappings”: {
    }
  }
}

As you can see from the response, the “mappings” section is empty, which implies there is no mappings applied at this stage.

Step 2

Now index a document to the created index, like below:

curl -XPUT localhost:9200/testindex-0202/testtype/1 — d ‘{
 “name”: ”ArunPMohan”,
 ”age”: 31,
 ”married”: true
}’

As you can see in the above document, I have three fields and the types of the fields are string (for the “name” field), integer (for the “age” field) and boolean (for the “married” field). When we type in this command, it results in no error, which simply means that Elasticsearch has auto detected the field types. The above command for indexing the document, yields the following output from Elasticsearch:

{
  “_index”: ”testindex-0202",”_type”:”testtype”,”_id”:”1",
  ”_version”: 1,
  ”result”: ”created”,
  ”_shards”: {
    “total”: 2,
    ”successful”: 1,
    ”failed”: 0
  },
  ”created”: true
}

Now as we have created the document, we can use the “GET mapping” API to see the mapping changes. So apply the following command:

curl -XGET localhost:9200/testindex-0202/_mapping

The response is as given below:

{
  “testindex-0202”: {
    “mappings”: {
      “testtype”: {
        “properties”: {
          “age”: {
            “type”: “long”
          },
          “married”: {
            “type”: “boolean”
          },
          “name”: {
            “type”: “text”,
            “fields”: {
              “keyword”: {
                “type”: “keyword”,
                “ignore_above”: 256
              }
            }
          }
        }
      }
    }
  }
}

In the above response, please note the “mappings” section. Here the first object under the mappings is the “testtype”. The “testtype” indicates the type of the index. Below the “testtype”, we have the “properties” object, which holds the list of fields and their type. You can see the “age” field is of the type “long”, the “married” field of the type “boolean” and the “name” field of the type “text”. The “text” type indicates the field is of the type “string”. Under the name field we have another section called as “fields”. We will see more about the “fields” section in the later blogs, for now, it is enough that it comes with the string/text field as default.

Things to know about Mappings

Let us have a look at some of the important things that would help us to understand more about mappings.

1. Updating mappings.

In an Elasticsearch there would be multiple types and for each type has its own mappings. Once a mapping is generated for the number of fields in under a type, it cannot be modified. That is if a document consists of 10 fields, 10 mappings are generated and we cannot modify it afterwards. In case if we want to modify it, we need to delete the index and then apply the modified mapping manually and then re-index the data.

Another important thing to be noted here is that, the above is true if we if try to modify the existing mapping, but for a new field, we are able to update the mapping. That is, if we add a 11th field to the document, in between, the mapping will auto-update accordingly. To make things much more clear, let us index a new document to the “testindex-0202”. Here in the new document, we are including a new field called “country”, apart from the “name”,”age” and the “married” fields. Let us do the same like below:

curl -XPUT localhost:9200/testindex-0202/testtype/2 -d ‘{
 “name”: ”AlexJohn”,
 ”age”: 32,
 ”married”: true,
 ”country”: ”Italy”
}’

Now hit the GET mapping API as we have done before. In the results you can see there is another field added called “country”.

2. Common Error

One of the most common error when dealing with mmappings is that the changing of the existing types of a field. Let us see this error for ourselves and have a better understanding. In the index “testindex-0202”, so far we have indexed 2 documents. Both the documents had the same type for the field “age” and the type was “long”. Now let us try to index a document like below:

curl -XPUT localhost:9200/testindex-0202/testtype/3 -d ‘{
 “name”: ”LexaMathew”,
 ”age”: ”twenty”,
 ”married”: false,
 ”country”: ”Italy”
}’

Note that here the age is given in string type, which indicates that it is a string field. The response for the above request will be like below:

{
  “error”: {
    “root_cause”: [
      {
        “type”: ”mapper_parsing_exception”,
        ”reason”: ”failedtoparse[
          age
        ]”
      }
    ],
    ”type”: ”mapper_parsing_exception”,
    ”reason”: ”failedtoparse[
      age
    ]”,
    ”caused_by”: {
      “type”: ”number_format_exception”,
      ”reason”: ”Forinputstring: \”twenty\””
    }
  },
  ”status”: 400
}

In the above response we can see the error “mapper_parsing_exception” on the field “age”. This indicates that the the expected field here was of another type and not string. In such cases re-index the document with the appropriate type or check for the data and see if there is any data modifications to be done and change data format/mapping accordingly.

3. Applying Custom mapping

So far we have seen the auto generation of mapping, that is mapping was generated by elasticsearch only. Now the most common use case is to apply our own mapping. This can be done by using the PUT mapping API in Elasticsearch like below:

First create an index of the name “testindex-0202a”

curl -XPUT localhost:9200/testindex-0202a

Now use the PUT mapping API like below:

curl -XPUT localhost:9200/testindex-0202a/_mapping/testtype -d ‘{
  “properties”: {
    “name”: {
      “type”: ”text”
    }
  }
}’

This would create a mapping with “name” as a string/text field. Now any documents indexed in the index “testindex-0202a” and the type “testtype” would be expected to have the “name” field of the type “text”.

4. Best practice in applying mappings

In practice what happens is that the data types for a field might vary in different points of time. So it takes us a couple of iterations for us to fix on the structure/type of the document fields. So, we might need to constantly change the data type/structure or mapping for the index. What usually is a recommended practice is to create a shell script, which will first delete the existing index and then apply mapping to the index. The advantage of this practice being, it will be easy to delete the existing index and then modify the mapping by creating a new index, thus saving us a lot of time.

Conclusion

In this blog we have covered the basics of Elasticsearch mappings like the application of mapping by Elasticsearch, some best practices and also how to apply custom mapping to an Elasticsearch index. In the next blogs we will see in detail about the Elasticsearch analysis and analyzers.