Java Flight Recorder and Kubernetes

When developing any production application, you always want to understand how the application is performing and what, if any, events are causing excessive latency, excessive memory, or CPU resources. If you are developing Java applications, then one of the better tools to investigate the behavior of your application is the Java Flight Recorder (JFR) tool. In addition to the JFR, the Java Mission Control (JMC) tool provides a graphical interface for viewing the output of the JFR.

The JFR is integrated into the Java Virtual Machine and causes almost no performance overhead. This means you can use the tool in heavily loaded production environments.

In this blog I will demonstrate how to use and capture a JFR when deploying your application code within a container and deployed to Kubernetes.

With the application deployed to Kubernetes you will want to get a shell to a running container. If there are multiple containers in the pod, then you will need to make sure you select the correct container to start the JFR. Let’s look at an example:

You can see that the todoserver-1 through 3 each have 4 containers per pod. We need to determine which container we want to start the flight recorder. If you aren’t familiar with the application, then execute a kubectl -n <namespace> describe pod <name of the pod> to identify each of the containers in the pod.

In our example: kubectl -n todolist-domain-ns describe pod todolist-domain-todoserver-1. Executing the command shows the following containers: istio-proxy, fluentd-stdout-sidecar, monitoring-exporter, and weblogic-server.

We are interested in monitoring the Weblogic server container. Therefore, we will get a shell in the container. This is done with the kubectl command,

kubectl -n todolist-domain-ns exec -it todolist-domain-todoserver-1 -c weblogic-server — jcmd

With the process identifiers located you can now start a flight recording for one of the processes. Again, you will select the weblogic.Server with process identifier 1781.

Now that we have the container and the process identifier determined you can now start a flight recording for this process. It can be done using kubectl and starting a shell within the container.

kubectl -n todolist-domain-ns exec -it todolist-domain-todoserver-1 -c weblogic-server — jcmd 1781 JFR.start name=myrecording filename=myrecording.jfr

There are several parameters that may be applied when starting a flight recording. These parameters will not be covered or discussed in this document. It is left to the reader to review these parameters.

Once you are satisfied with the amount of time the flight recording has been running then you are able to stop the recording or dump the recording. Review the documentation to understand the difference.

The flight recording has been captured and recorded. Keep in mind that the recording is written within the container’s filesystem. You need to bring that recording to a machine that has the Java Mission Control (jmc) utility installed. This is accomplished with another kubectl command to gain access to a shell for the container.

The format of the command is outlined below:

kubectl <command> <namespace>/<pod name>:<filename> <local filename> -c <container>

With the flight recorder file on the local machine, you are now free to load the flight recorder into the Java Mission Control application.

Summary

Even though the Java services are being containerized and ultimately deployed to Kubernetes does not mean you have to give up using one of the more functional tools for monitoring the Java services. Using kubectl to gain a shell into the container allows you to start, stop, and dump flight recordings. These recordings can be copied to a machine and eventually loaded into the JMC.

Having the flight recorder loaded into the JMC you are now able to evaluate garbage collections, running threads, blocked threads, hot methods, CPU utilization, and many other events and metrics.

If you are experiencing any issues with your Java services, it is advisable to capture some flight recordings. These recordings will jumpstart your efforts in identifying and resolving those issues.