ForgeRock OpenDJ/Directory Services Performance Tuning
OpenDJ also known as Directory Services is ForgeRock’s very own directory server a fork of OpenDS and implements a wide range of LDAP related standards including full compliance with LDAPv3. In this article, we’ll go over some of the tuning parameters that can help OpenDJ achieve performance. Most of these parameters are found in backstage documentation and a few are based on personal experience working with the product.
I cannot stress this enough but tuning a JVM application like OpenDJ is an ongoing process and must be reevaluated if there’s a change in internal or external factors. At minimum, you want to revisit tuning every quarter and see how it’s performing and whether it requires any further tweaks.
Java Virtual Machine (JVM) Tuning
Garbage Collection (GC) is a major part of JVM tuning and for that we need to understand the current GC activity. To do this, we will log GC data to a separate log file for further analysis. And below are the parameters we will use to output GC data:
// Prints details about promotion failure
-XX:+PrintPromotionFailure// Prints free list statistics for each Young/Old collection
-XX:PrintFLSStatistics=1// Prints threshold and ages of objects in New Generation
-XX:+PrintTenuringDistribution// Prints GC details
-XX:+PrintGCDetails// Prints GC timestamp and datetime details
-XX:+PrintGCTimeStamps
-XX:+PrintGCDateStamps// Prints cause of GC
-XX:+PrintGCCause// Outputs data to a separate log file
-Xloggc:[FILENAME]// # of log GC files, depends upon use case
-XX:NumberOfGCLogFiles=10// Rotate GC files
-XX:+UseGCLogFileRotation// Log file size - Keep this small for portability
-XX:GCLogFileSize=5m
I would recommend running your DJ with the above parameters for a few days in order to capture as much data possible. And once you have the data, I would suggest running it through a tool like “gceasy.io” or an offline tool of your choice. Once you are done with capturing GC data, don’t forget to tune these parameters down to avoid excessive logging.
Having done similar analysis gazillion times, I would recommend trying the below JVM parameters. And just so that we are clear - tuning is not one size fit all approach and highly varies for every use case. I would highly recommend not to solely rely on these parameters and duly test your configuration in a lower environment before promoting to a production environment.
// Concurrently runs GC while application threads are in RUNNABLE state
-XX:+UseConcMarkSweepGC// Disables any explicit GC invocation. E.g. System.gc()
-XX:+DisableExplicitGC// Enables use of compressed 32 bit OOPS in 64 bit JVM to effectively improve CPU cache efficiency
-XX:+UseCompressedOops// Kick-in CMS when Old Generation is 75 percent full
-XX:CMSInitiatingOccupancyFraction=75// To be used in conjunction with above parameter
-XX:+UseCMSInitiatingOccupancyOnly// Instructs to run collection in Young before doing Full GC
-XX:+ScavengeBeforeFullGC// Instructs to run collection in Young before CMS remark phase
-XX:+CMSScavengeBeforeRemark// Min heap size - Tweak as necessary
-Xms6g// Max heap size - Tweak as necessary
-Xmx8g// Young Generation Max Size - Tweak as Necessary
-XX:MaxNewSize=3g// Young Generation Min Size - Tweak as Necessary
-XX:NewSize=3g// Perm Generation - this should suffice most requirements
-XX:MaxPermSize=256m
Young and Old Generation
If you have set the values of -Xmsand -Xmx to be identical — I would recommend testing the opposite. Setting the values to be same — disables the “intelligent” behaviour of JVM to make memory recommendations for itself. For example, let’s say you have set both the parameters to 8 GB but what if JVM doesn’t need 8 GB at all times. What if the peak usage on certain days/time is only 6GB. In that case, running GC against 8 GB would take more time as compared to 6GB. The greater the heap size more time it will take to run the garbage collection.
Now as mentioned earlier, there’s no hard set rule and you will need to try and test this out in your environment. I would recommend running a benchmark against your DJ in a lower environment and collect 75 percentile and peak usage. I would then set the -Xms to the value of 75th percentile value and -Xmx with the peak usage. You may also want to take into consideration any internal/external factors specific to your usage.
DB Cache
Continuing into the tuning, it is also important to look into memory requirements set for the database cache. The feature allows DJ to cache backend data in-memory for faster response times. When the backend cannot be fully cached, the DB needs to access records from log files. Since only a portion of the files can be opened at a time, the DB keeps closing them and opening the ones it needs. Each close does a disk sync, which has a large penalty on performance. So in order to avoid this to occur, we need to allocate and tweak the memory allocated towards the database cache. Now prior to version 6.5, the DB cache size was set individually for each JE backend using the property db-cache-percent . This has changed in version 6.5 where a shared cache is used instead by default. I would recommend going through the official DJ documentation on this topic.
VM Swap (Optional)
If you notice your VM is constantly swapping GBs of data, I would recommend look into tweaking the swapinessvalue of your VM. If you happen to have lots of extra memory available, I would suggest tweaking the default swapiness value from 40 to 20 . Setting the value to 20 just means, the Linux Kernel will start swapping or using disk when RAM is 80% full. This would increase the overall performance of your VM since the early use of disk is delayed until RAM is 80% full.
On-Going Initiative
To wrap up — I would again say that tuning a JVM application is an on-going initiative and you will have to revisit this topic at least every quarter if not more. It’s like turning a bolt using a ranch — you will need to know which ranch to use, how much pressure to apply, which direction to turn, and will need to revisit when it loosens again. That is all for now on the tuning and if you have any questions or feedback, please leave it in the comment section.
