Elasticsearch 6.0 Removal of mapping types

series : Elasticsearch 6.0 is coming

Image for post
Image for post
Disclaimer : Elasticsearch 6.0 is not yet released so please use it carefully for sure NOT in a production environment. This article is intended for testing and understanding the new improvements from ES 6.0. 

What are Elasticsearch Types ?

Image for post
Image for post
Image for post
Image for post
GET articles/_search
{
“query”: {
“match”: {
“user_name”: “kimchy”
}
}
}
GET articles/user,blogpost/_search
{
“query”: {
“match”: {
“user_name”: “kimchy”
}
}
}
GET articles/blogpost/_search
{
“query”: {
“match”: {
“user_name”: “kimchy”
}
}
}
Image for post
Image for post
user_name is the same field between user and blogpost Type.
{
"root_cause": [{
"type": "illegal_argument_exception",
"reason”: "mapper [user_name] cannot be changed from type [text] to [integer]"
}],
"type": "illegal_argument_exception",
"reason": "mapper [user_name] cannot be changed from type [text] to [integer]"
}

Why Types have been removed ?

Index : productsType : item
field1 : 'model_id' - type : integer
field2 : 'name' - type: text
field3 : 'country' - type: text
field4 : 'country_name' - type: text
field5 : 'color' - type : array
field6 : 'date' - type : date
Type : item_price
field1 : 'model_id' - type : integer
filed2 : 'price' - type : float
field3 : 'model_discount' - type : integer
field4 : 'currency' - type : text
Image for post
Image for post
Type : item
Image for post
Image for post
Type : itemp_price
Image for post
Image for post
Visualise Sparsity
For both NUMERIC fields and ordinals of SORTED fields, we store data in a dense way. As a consequence, if you have only 1000 documents out of 1B that have a value, and 8 bits are required to store those 1000 numbers, we will not require 1KB of storage, but 1GB.I suspect this mostly happens in abuse cases, but still it’s a pity that we explode storage requirements. We could try to detect sparsity and compress accordingly.
Image for post
Image for post
Sparse Lucene benchmarks
Image for post
Image for post

How can I use Type in ES 6.0 ?

Image for post
Image for post

Testing Elasticsearch 6.0

curl -XDELETE localhost:9200/products
curl -XPUT 'localhost:9200/products?pretty' -H 'Content-Type: application/json' -d'
{
"mappings": {
"item": {
"properties": {
"model_id": {
"type": "integer"
},
"name": {
"type": "text"
},
"country": {
"type": "integer"
},
"country_name": {
"type": "text"
},
"color": {
"type": "text"
},
"date": {
"type": "date"
}
}
},
"item_price": {
"properties": {
"model_id": {
"type": "integer"
},
"price": {
"type": "float"
},
"model_discount": {
"type": "float"
},
"currency": {
"type": "text"
}
}
}
}
}
'
{
"products": {
"aliases": {},
"mappings": {
"item": {
"properties": {
"color": {
"type": "text"
},
"country": {
"type": "integer"
},
"country_name": {
"type": "text"
},
"date": {
"type": "date"
},
"model_id": {
"type": "integer"
},
"name": {
"type": "text"
}
}
},
"item_price": {
"properties": {
"currency": {
"type": "text"
},
"model_discount": {
"type": "float"
},
"model_id": {
"type": "integer"
},
"price": {
"type": "float"
}
}
}
},
"settings": {
"index": {
"creation_date": "1506334542346",
"number_of_shards": "5",
"number_of_replicas": "1",
"uuid": "VgmCm2CbQZyS0ZtDz4GpTA",
"version": {
"created": "5060199"
},
"provided_name": "products"
}
}
}
}
curl -XPOST 'localhost:9200/_bulk?pretty' -H 'Content-Type: application/json' -d'
{"index":{"_index":"products","_type":"item","_id":1}}
{"model_id" : 33, "name" : "model1", "country" : 1, "country_name" : "it_IT", "color": "red", "date" : "2017-09-21"}
{"index":{"_index":"products","_type":"item","_id":2}}
{"model_id" : 34, "name" : "model2", "country" : 1, "country_name" : "it_IT", "color": "red", "date" : "2017-09-21"}
{"index":{"_index":"products","_type":"item","_id":3}}
{"model_id" : 35, "name" : "model3", "country" : 2, "country_name" : "de_DE", "color": "yellow", "date" : "2017-09-20"}
{"index":{"_index":"products","_type":"item","_id":4}}
{"model_id" : 36, "name" : "model4", "country" : 1, "country_name" : "es_ES", "color": "green", "date" : "2017-09-19"}
{"index":{"_index":"products","_type":"item","_id":5}}
{"model_id" : 37, "name" : "model5", "country" : 1, "country_name" : "it_IT", "color": "yellow", "date" : "2017-09-21"}
'
curl -XPOST 'localhost:9200/_bulk?pretty' -H 'Content-Type: application/json' -d'
{"index":{"_index":"products","_type":"item_price","_id":1}}
{"model_id" : 33, "price" : "9000", "model_discount" : 10, "currency" : "euro"}
{"index":{"_index":"products","_type":"item_price","_id":2}}
{"model_id" : 33, "price" : "5500", "model_discount" : 0, "currency" : "euro"}
{"index":{"_index":"products","_type":"item_price","_id":3}}
{"model_id" : 33, "price" : "16000", "model_discount" : 0, "currency" : "us dollar"}
{"index":{"_index":"products","_type":"item_price","_id":4}}
{"model_id" : 33, "price" : "3500", "model_discount" : 4, "currency" : "english pound"}
{"index":{"_index":"products","_type":"item_price","_id":5}}
{"model_id" : 33, "price" : "8970", "model_discount" : 3, "currency" : "euro"}
'

Create a snapshot fro products index :

curl -XPUT localhost:9200/_snapshot/my_backup -H 'Content-Type: application/json' -d '{
"type": "fs",
"settings": {
"location": "my_backup",
"compress": true
}
}'
curl -XPUT 'localhost:9200/_snapshot/my_backup/snapshot_1?wait_for_completion=true&pretty' -H 'Content-Type: application/json' -d '{
"indices": "products",
"ignore_unavailable": true,
"include_global_state": false
}
'
curl -XPUT localhost:9200/_snapshot/my_backup -H 'Content-Type: application/json' -d '{
"type": "fs",
"settings": {
"location": "my_backup",
"compress": true
}
}'
curl -XPOST localhost:9200/_snapshot/my_backup/snapshot_1/_restore?pretty

How to upgrade multiple type Indices to just one type.

Split your index in a single type Index:

Image for post
Image for post
Reindex API — the army knife you have to use for updating your index
yellow open products iP3eHQ7lS4CBrcTjROMCEA 5 1 10 0 32.6kb 32.6kb
curl -XPOST 'localhost:9200/_reindex?pretty' -H 'Content-Type: application/json' -d'
{
"source": {
"index": "products",
"type": "item"
},
"dest": {
"index": "item"
}
}
'
{
"item" : {
"aliases" : { },
"mappings" : {
"item" : {
"properties" : {
"color" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
...
...
yellow open products  ...
yellow open itemprice ...
yellow open item ...

Use a custom field and filter by that field to discriminate a type:

geoshapes:
types:
administrative_level_1
name:
{ type: string }
slug: { type: string, index: not_analyzed }
polygon:
type:
geo_shape
tree: quadtree
precision: 50m
center: { type: geo_point }
population: { type: integer, index: not_analyzed }
administrative_level_2:
name:
{ type: string }
slug: { type: string, index: not_analyzed }
polygon:
type:
geo_shape
tree: quadtree
precision: 50m
center: { type: geo_point }
population: { type: integer, index: not_analyzed }
administrative_level_3:
name:
{ type: string }
slug: { type: string, index: not_analyzed }
polygon:
type:
geo_shape
tree: quadtree
precision: 50m
center: { type: geo_point }
population: { type: integer, index: not_analyzed }
geoshapes:
administrative_level:
type:
{ type: string }
name: { type: string }
slug: { type: string, index: not_analyzed }
polygon:
type:
geo_shape
tree: quadtree
precision: 50m
center: { type: geo_point }
population: { type: integer, index: not_analyzed }

Conclusion

References

Written by

Entrepreneur, Coder, Head of Development @motorK