How To Configure Elasticsearch Heap Size to Change Max Memory Size

It is a very important setting for ElasticSearch. In order for this setting to be set correctly, the system must be well monitored. In short, we determine the amount of memory that Elasticsearch will allocate at the beginning and maximum memory usage, with this config.

Before configuring Heap Size, I will talk about the heap memory concept to monitor heap size correctly. Java applications use the “stack” and “heap” methods to save data to memory.

In Stack type, the operating system manages whether the data will be stored or deleted in memory.

In Heap type, the application manages memory usage and cleaning. In Java, objects are stored in the heap memory and thanks to the structures called Garbage Collector, heap memory management is provided.

What is Garbage Collector?

In Java, memory management is done in the background with JVM and the Garbage Collector included in it. Garbage Collector simplifies application development and shortens coding time with automatic memory management. Again, Garbage Collector is able to deal with the problem of leaking memory areas caused by coding errors.

Garbage Collection is an automated memory management mechanism. it looks at Heap memory, detects used objects and deletes non-referenced ones. The space occupied by unused / non-referenced objects is cleared from memory and this increases available free memory. The mechanism that performs this process is called Garbage Collector.

Excessive heap size can be used for caching, but excessive heap size can cause pauses on the garbage collector. Return to end-user can be very long when GC stops.

When the long pause is experienced, access does not occur in distributed systems such as Elasticsearch, as the node may be isolated from the cluster. There is no reading or writing operation on this node. If the node is a Master node, a new Master node can be selected. If the node is data node, it may cause the shards to be placed in other data nodes. In this case, it increases the network traffic, the input-output operations on the disk and the load of the cluster.

Briefly, if heap size is set to be less than you need, other problems may be encountered besides memory errors.

Some Best Practice Notes;

• It is highly recommended that Heap size not be more than half of the total memory. So if you have 64 GB of memory, you should not set your Heap Size to 48 GB.
• Heap Size is not recommended to exceed 32 GB.
• In Default, Xms1g and Xmx1g is 1 GB.
• To change the JVM heap size, the default path address is /etc/elasticsearch/jvm.options.

Apart from these, you can share other factors that should be considered, as comments.

Thank you

Good luck with.