JVM OOM in Kubernetes POD with small memory allocated
How to configure JVM memory on POD with a small memory resource.
Problem
A Kubernetes POD is configured with max 1000 MB memory and 1 CPU. Recently high memory usage has been observed(90%+) and results in POD being killed by OOM. In this article, I will explain how I solve this issue.
Tools
The first thing is we need to understand the application’s current memory usage. We can use a range of tools to track the memory and heap usage. Refer to https://www.baeldung.com/java-heap-dump-capture. However, there are some restrictions.
GUI
jconsole and visualvm are powerful tools to monitor memory usage, but in a Kubernetes cluster environment, usually we don’t have permission to connect to remote JVM/JMX port.
Command line tool
In case we can SSH to POD and have JDK tools installed, we can use jstat to print the memory usage.
# pid is 1 in POD, print every 10000 ms
jstat -gc -t 1 10000
S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT GCT
512.0 512.0 0.0 32.0 31232.0 18740.0 175104.0 304.1 4864.0 2435.9 512.0 266.9 297 0.591 0 0.000 0.591
512.0 512.0 0.0 64.0 31232.0 19989.0 175104.0 304.1 4864.0 2435.9 512.0 266.9 301 0.591 0 0.000 0.591
512.0 512.0 32.0 0.0 31232.0 4372.6 175104.0 304.1 4864.0 2435.9 512.0 266.9 310 0.603 0 0.000 0.603
512.0 512.0 32.0 0.0 31232.0 13117.7 175104.0 304.1 4864.0 2435.9 512.0 266.9 314 0.609 0 0.000 0.609and use jcmd to retrieve heap dump:
./jcmd 1 GC.heap_dump /path/to/heapdump-file
# run the below command at local machine, to copy heapdump fiel to local machine
kubectl cp mypod-name:/path/to/heapdump-file ./local-file-name -n namespace
# or print heap info
./jcmd 1 GC.heap_info
# or print native memory statistic(must enable native memory summary in java command line)
./jcmd 1 VM.native_memory summaryObservability
In Cloud native applications, we often use Spring actuator, Jolokia, micrometer, and Prometheus to produce the application JVM metrics, On the other side we use collectors such as Telegraf, Prometheus and inject metrics into sumologic, datadog, etc.
GC log
we can also add this to the Java application command line to enable GC activities log at runtime.
"-Xlog:gc*=info"
# or mode detail
"-Xlog:gc*=debug"example =>
2021-10-07T14:13:10.847+0800: [Full GC (Metadata GC Threshold) [PSYoungGen: 384K->0K(93696K)] [ParOldGen: 982K->1360K(1172480K)] 1366K->1360K(1266176K), [Metaspace: 410659K->410659K(456704K)], 0.0418110 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]Use Code
or we can use the below code to get more pool details. e.g. direct pool CodeCache Compressed Class Space etc. it will print the statistic every 10 seconds. The different memory categories will be explained next.
MemoryMonitor.kt
JVM memory configuration
Dockerfile
FROM azul/zulu-openjdk:17.0.7-17.42.19-jre
ARG DEPENDENCY=build/dependency
COPY --chown=userXX:userXX ${DEPENDENCY}/BOOT-INF/lib /app/lib
COPY --chown=userXX:userXX ${DEPENDENCY}/META-INF /app/META-INF/
COPY --chown=userXX:userXX ${DEPENDENCY}/BOOT-INF/classes /app/
ENTRYPOINT ["java", \
"-server", \
"-cp", "/app:/app/lib/*", \
"-XX:MaxRAMPercentage=60.0", \
"-XX:+UnlockDiagnosticVMOptions", \
"-XX:NativeMemoryTracking=summary", \
"-XX:+PrintNMTStatistics", \
"com.company.myapp.ApplicationKt", \
"--spring.config.location=classpath:/application.yaml"]
The above Dockerfile uses a max of 60% RAM(e.g. we configured a total of 1 GB for POD) and enables native memory tracking. which is equal to 600 MB for heap usage, and around the remaining 400 MB for other usage(e.g. non-heap, JVM itself, and OS), look good? actually, 400 MB is far less than enough in nowadays modern cloud-native applications.
memory usage category
This table lists several important memory categories.
Metaspace, formerly known as PermGen (Permanent Generation), is a non-heap memory area that stores class metadata, constant pool information, and method bytecode. It was introduced in Java 8 as a replacement for PermGen, which was removed due to memory management issues.
It is unlimited if not specified. my observation is it can easily hit 200–300MB in spring boot application
Direct Buffer, is also unlimited, used in socket, nio. In general Spring application usually occupies just a small amount of memory(20~40MB), but in some Big data applications such as Spark, it may use a huge amount and cause OOM!
with the above Dockerfile setting, CodeCache and CompressedClassSpace limits are 256 MB and 64 MB respectively, the actual used will be <50 MB together.
Solution
We need to be aware that JVM and OS itself also have memory overhead. I would say it is about 25~30% of total memory. So the total memory consumption :
heap(600 MB) + metaspace(250 MB) + codecache and compressed_class_space(50 MB) + OS&JVM(300 MB,30% of total) = 1200 MB, which exceeds the POD 1 GB limit!
In this case, it is not caused by a memory leak, hence we need to calculate and allocate memory accordingly.
- increase the POD limit to 2 GB, this is a safer and easiest way
- lower the
MaxRAMPercentageto 40.0%, and/or limit the other non-heap usage, you can tweak the number according to what you observe in the log:
"-XX:MaxRAMPercentage=40.0", \
"-XX:MaxMetaspaceSize=176m", \
"-XX:CompressedClassSpaceSize=32m", \
"-XX:ReservedCodeCacheSize=48m", \
"-XX:MaxDirectMemorySize=24m",Hence we get:
heap(400 MB) + non-heap(280 MB: metaspace + codecache+compressed class…) => we left 320 MB room for OS/JVM, barely enough :)
One more thing, Thread dump analytics
Recently I encountered another thread leak issue: https://github.com/harness/ff-java-server-sdk/pull/107
If the thread count(from the above code) increases dramatically, we may want to analyze the thread dump.
jstack -l 1 > thread-dump-1.txt
# copy to local machine
# wait until the thread count changes significantly
jstack -l 1 > thread-dump-2.txtI use this online tool to compare the dump: https://jstack.review/
refer to https://www.baeldung.com/java-analyze-thread-dumps
Happy Coding!
