Heap Configuration in Elasticsearch
Heap is a memory area that object is stored in runtime. All of java objects are referenced in heap. The size of heap is maximized by application at startup. If the application’s references reach the size of heap, in other word heap is full, garbage collector gets involved in those situations. While the garbage collector is running, stop-the-world phase is activated. At this phase, all operations wait to finish the garbage collector’s job. The heap size affects stop the world phase, garbage collector’s duration, allocation, and reference speed. For these reasons, correct size of the heap is very important.
Nothing Less Nothing More
It should not consider that if the size of heap is large, every reference of objects are allocated memory. If the size of heap is larger than it should be, when heap is full and the garbage collector is started to work, stop-the-world phase’s duration will too long. In Elasticsearch long pauses caused that node is unreachable, all operations are suspended. Codes are not executed, none of system’s functions are called. If unreachable node is master, then workers appoint a new master node, it increases network and disk I/O traffic. If the non-responsive node is a data node, the master selects a new node to allocate the paused node’s assigned shards.
On the other hand, the heap size is too small, the application faced with the out-of-memory problem. The memory is insufficient for allocated new objects references. It also causes that short pauses for clearing heap. Short pauses have negative effects to number of indexing operation and reduce number of running queries.
Lucene has an insatiable desire for memory in Elasticsearch. Indices and docs are stored in segments of Lucene. Each segment is its own index. Lucene searches all of them in sequence. When new documents added to the index, they are added to the next segment. Previous segments are never modified. It means that segments are immutable. For this reason, these segments are very cache friendly and Lucene is another major user of memory beside heap. Elasticsearch recommends that never give more than half of total memory to heap.
There are two thresholds for the heap size, according to the architecture of the operation system. For understand the thresholds, it is necessary to figure out that how JVM points to the objects.
Compressed Ordinary Object Pointers (Compressed oop)
In 64-bit systems use word alignment (also known word addressing, 8 byte addressing). With 8 byte alignment, 3 lower bits are zero.
Pointers can be decoded from a 32-bit object offset. In other words, instead of pointing 4 billion bytes with 32-bit, 4 billion objects can be pointed. Java uses a data structure called ordinary object pointer(oop) that holds an object. In 64-bit systems for getting maximum performance from this addressing, use compressed oops.
JVM knows that the system uses 8 byte alignment and also knows that always lowest 3 bits are zero. There is no point for holding the last 3 bits. JVM shift last 3 zero bits, and add 3 more meaningful bits. Actually instead of 32 bit, 32 + 3 = 35 bit is stored in heap.
According to these calculations, do not exceed the 32GB limit. JVM instead of using object offset, it uses byte offset when the heap size larger than 32GB upper bound or less than 4GB lower bound. In example, around the 50–60GB of the allocated heap is less effective than the 32GB of heap with using compressed oops.
In conclusion, exceeding 32GB of heap size can cause that wasting memory, GC has difficulty to struggle the large heap and also reduces the CPU performance. Physically having more than 64GB Ram, give 32GB RAM to the heap, Lucene will take the whole remaining memory happily.
