ElasticSearch: Performance Tuning

ElasticSearch is a great tool for log analysis, commonly used with stack of Logstash and Kibana known as ELK stack.
Elasticsearch is highly utilise memory(RAM) of machine to perform the operation, also all the operations are highly I/O intensive, which increased load on disk and N/w. Recommended to use the SSD disks over HDD to boost disk performance and I/O operations.

Why need tuning?
As your infrastructure grows your systems start producing more and more logs, as log size and volumes grows, your ES performance stars degraded. For some time you have easy solutions like vertical scales, but thats causes more cost.
Afterward it’s comes to tuning and optimise ES performance.

How to handle 10k/sec requests?
For large systems log volume and data size is huge, so how we gonna handle the such high load. First step choose right size of machine as per need. Prefer memory optimized machine over cpu, because we need more RAM than CPU’s.
We are using r4.xlarge(30.5 Gb, 4 core)* 3 machine cluster for 3k/second requests.

Cluster setup
Setup at least 3 node cluster.And follow below steps-

Please update elasticseach.yml with configuration as below.

cluster.name: ES-cluster
 node.name: ${HOSTNAME}
 path.data: /data
 path.logs: /data/elasticsearch-service-logs
 ## Enable memory lock
 bootstrap.memory_lock: true
 network.host: <private_ip>
 discovery.zen.ping.unicast.hosts: [<node01_private_ip or hostname>, <node02_private_ip or hostname>, <node03_private_ip or hostname>]
 discovery.zen.minimum_master_nodes: 2
 thread_pool:
 bulk:
 queue_size: 6400
 index:
 queue_size: 6400

NOTE: Please update < private_ip’s > field.

Adjusting queue size

ElasticSearch processes maintain queues for processing requests, it’s important to set right queue size.By default queue size is 100.
If your queue is full ES starts denied request from logstash.

Sample logstash logs:

[2017–07–05T18:40:21,052][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 429 ( {“type”=>”es_rejected_execution_exception”, “reason”=>”rejected execution of org.elasticsearch.transport.TransportService$6@779c8dbc on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@693ef756[Running, pool size = 2, active threads = 2, queued tasks = 50, completed tasks = 1161414]]”})

Try to adjust best queue size.

Configure JVM
Minimum setting for jvm heap size.Assign at least 50% memory of total RAM available, should not exceed 32 GB.

Update jvm.options as below.

-Xms16g
-Xmx16g

Update system file size and descriptors

Edit `/etc/default/elasticsearch` as below.

ES_HEAP_SIZE=16g
MAX_OPEN_FILES=99999
MAX_LOCKED_MEMORY=unlimited

Setup index rotation
For index rotation use curator plugin, for install and setup instruction use this link:

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html

Sample config files: curator.yml

— -
 client:
 hosts:
 — elknode01
 port: 9200
 url_prefix:
 use_ssl: False
 certificate:
 client_cert:
 client_key:
 ssl_no_validate: False
 http_auth:
 timeout: 30
 master_only: False
 
 logging:
 loglevel: INFO
 logfile:
 logformat: default
 blacklist: [‘elasticsearch’, ‘urllib3’]
 — -
 `cluster_delete_index.yml`
 
 actions:
 1:
 action: cluster_routing
 description: >-
 Disable shard routing for the entire cluster.
 options:
 routing_type: allocation
 value: none
 setting: enable
 wait_for_completion: True
 disable_action: False
 2:
 action: delete_indices
 description: >-
 Delete indices older than 8 days (based on index name), for logstash-
 prefixed indices. Ignore the error if the filter does not result in an
 actionable list of indices (ignore_empty_list) and exit cleanly.
 options:
 ignore_empty_list: True
 disable_action: False
 filters:
 — filtertype: pattern
 kind: prefix
 value: logstash-
 — filtertype: age
 source: name
 direction: older
 timestring: ‘%Y.%m.%d’
 unit: days
 unit_count: 8
 3:
 action: cluster_routing
 description: >-
 Re-enable shard routing for the entire cluster.
 options:
 routing_type: allocation
 value: all
 setting: enable
 wait_for_completion: True
 disable_action: False

Add cron

Setup cron job:
00 12 * * * /usr/local/bin/curator ~/.curator/cluster_index_delete.yml

Summary

ElasticSearch tuning is add values into production and huge benefits to keep cluster smooth and healthy.

ElasticSearch tuning is very important and critical task as it can break your cluster, so be careful while modifying any parameters.

Hope this post help you to add some good tactics in tuning and happy Searching.