Elasticsearch under Shenandoah

How you may know I’m in love with Shenandoah. Here I’d like to share with you some results over running elastic search under Shenandoah.

This experiment was made on unnamed cluster with dozens of machine and I’d like to share only two pictures. One from node with enabled Shenandoah for last week, and second one has G1 with settings closer to that I described before. Both machine has very similar number of segments and this cluster uses almost for writing. Let say 99% of operation it is writing.

Summary: at peaks loading it is much better and at night it is similar. So, it is expected behaviour to be honest. Also it removes full GC that brokes this cluster time-to-time.

You can see some result at the graph for last 24 hours:

Image for post
Image for post
G1
Image for post
Image for post
Shenandoah

But it has two issues:

  1. Internal metrics for GC counts doesn’t work at all.
  2. Shenandoah makes a lot of things as separated non blocking threads and it brokes logic of JvmGcMonitorService, and logs full of lines like:
[2018-12-26T12:57:40,974][WARN ][o.e.m.j.JvmGcMonitorService] [es9-main] [gc][422543] overhead, spent [770ms] collecting in the last [1s][2018-12-26T12:57:50,976][WARN ][o.e.m.j.JvmGcMonitorService] [es9-main] [gc][422553] overhead, spent [1.3s] collecting in the last [1s][2018-12-26T12:57:58,978][WARN ][o.e.m.j.JvmGcMonitorService] [es9-main] [gc][422561] overhead, spent [929ms] collecting in the last [1s][2018-12-26T12:58:06,979][WARN ][o.e.m.j.JvmGcMonitorService] [es9-main] [gc][422569] overhead, spent [946ms] collecting in the last [1s][2018-12-26T12:58:15,981][WARN ][o.e.m.j.JvmGcMonitorService] [es9-main] [gc][422578] overhead, spent [749ms] collecting in the last [1s]

Costs? Well, it increases CPU usage but it is expected behaviour.

Written by

IT geek who loves to play with the data. Would like to contact me? Just drop an email to kirill@korins.ky

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store