Remotely connecting through Kubernetes to a JMX agent in a Spring Boot 1.x application to monitor an application

Tremaine Eto

Follow

Oct 31, 2018 · 6 min read

What‘s JMX, Kubernetes, and Spring Boot?

The Java Management Extensions (JMX) technology is a Java standard and allows for a means of monitoring and managing resources–most notably the Java Virtual Machine (JVM).

A resource is instrumented with Managed Beans (MBeans) that are registered in a core-managed MBean server which can run on devices that have the Java programming language installed.

Kubernetes is an open-source container-based system that automates deployments, scaling, and general container management. In a use case relevant to this article, it can orchestrate Java applications that are deployed in Docker containers.

Spring Boot is a Java framework for running a Spring application to support stand-alone, production-grade applications. It includes features such as built-in annotations and embedded HTTP servers like Tomcat or Jetty.

What’s the problem?

As it turns out, connecting to a JMX agent locally through a JMX UI client is quite simple. Typically, the client is automatically aware of the application through its running .jar file and gives the user the option to connect to its JMX port, which in Spring and in many other cases is by default 1099.

However, once a Spring Boot application is deployed in Kubernetes, the connection somehow ceases to work, at least in my case when I tried doing so.

After searching for possible solutions online, it became clear that this issue was somewhat common, as evidenced by multiple threads on Stack Overflow. Unfortunately, none of the solutions provided seemed to directly work for me, and none of them seemed directly tailored to Kubernetes specifically.

A large part of the issue is that in a remote connection through a typical RMI connector, then a second random port is essentially chosen for a user’s application as an accessible JMX endpoint. It’s essentially impossible for someone to dynamically be able to guess this endpoint. This blog post by Mark Feeney goes so far as to say “(Don’t Use) RMI” — instead, he (among others) suggests using the JMX Messaging Protocol, or JMXMP.

This article will just step through the process I went through for getting a remote JMX connection to work through Kubernetes via JMXMP; it is likely not the only possible solution to the issue but hopefully will add to the overall solution set.

Changes you need to make in your Spring Boot application

For this approach to work, you need to make a few changes in your Spring Boot application itself.

The first of which is to create a class that exposes a new @Bean; this bean will have to be a ConnectorServerFactoryBean, and it essentially needs to have its service URL set to a dynamic JMX URL that is forced onto port 1099 instead of a random port. A code example is as follows.

@Bean
public ConnectorServerFactoryBean connectorServerFactoryBean() throws Exception {
   final ConnectorServerFactoryBean connectorServerFactoryBean = new ConnectorServerFactoryBean();
   String jmxURL = "service:jmx:jmxmp://localhost:1099/";
   connectorServerFactoryBean.setServiceUrl(jmxURL);   return connectorServerFactoryBean;
}

As you can see, the server URL that you would be able to access the JMX agent is now through the JMXMP protocol: service:jmx:jmxmp://localhost:1099/.

Additionally, ensure that the following property is set in the application.properties (or application.yml) file of your Spring Boot application:

spring.jmx.enabled=true

Now, deploy your Spring Boot application to Kubernetes through your usual process.

Changes you need to make in Kubernetes

Essentially, you just need to make sure you (or your Kubernetes administrator(s)) open up and thus expose the 1099 port through a NodePort for your particular deployment. This can be done by editing the deployment’s Service resource’s YAML file and adding the port to the spec object; there is some documentation online on how to do so.

Ultimately, you will access the JMX agent through this NodePort as the port and through the pod’s NodeIP as the IP.

How to use the JMX clients to connect

Firstly, you’re going to need to download a special JAR file called opendmk_jmxremote_optional.jar to your local machine. This JAR file will enable you to connect to a JMX agent through JMXMP. There are different download options, and I’ve found MvnRepository to be a trust-worthy source (as always, exercise caution when downloading files online).

When using any JMX client (such as JConsole, Java Mission Control, VisualVM, etc.), I’ve found that you must specify the location of your opendmk_jmxremote_optional_jar JAR file as an argument or by adding it to your Java installation’s “endorsed” directory. Below are the steps to do each method.

Method 1: Specifying endorsed directory with JMX client

JConsole

1. Change directory (cd) to the directory where your OpenDMK JMXRemote JAR resides in
2. $ jconsole ‘-J-Djava.endorsed.dirs=.’
3. You will see this pop up (if not, you may need to install JConsole):

4. Under “Remote Process”, insert the following: service:jmx:jmxmp://<KubernetesNodeIP>:<KubernetesNodePort>/

5. A successful connection should be made and the necessary MBeans and metrics should be viewable

(J)VisualVM

1. Change directory (cd) to the directory where your OpenDMK JMXRemote JAR resides in

2. $ (j)visualvm (dash dash) cp:a path/to/your/opendmk_jmxremote_optional_jar

Note: you can add the “j” if you want to use JVisualVM. Also, replace the (dash dash) with two hyphens; Medium automatically converts this to an em dash, which is not what you want to use.

3. This will pop up; click the button circled:

4. In the “Connection” box, enter service:jmx:jmxmp://<KubernetesNodeIP>:<KubernetesNodePort>/

5. Click “Do not require SSL connection”

6. Click OK

7. Under “Remote”, you will now see the KubernetesNodeIP show up; you’ll be able to doubleclick on the service URL and see the metrics and MBean information

Method 2: Setting up local Java endorsed directory

Being in a particular folder that the JMX JAR file resides in can get cumbersome, and thus it may be beneficial to set up a local Java endorsed directory with the JAR file inside. Here are the steps to do so.

1. $ cd path/to/your/jdk1.x.x_xx/lib
   ( To find your machine’s JDK installation path, you can run $whereis java and then $ cd /lib)
2. $ mkdir endorsed
3. $ cd endorsed
4. $ cp path/to/your/opendmk_jmxremote_optional_jar-1.0-b01-ea.jar .
   (Don’t forget the period at the end! This specifies the current directory as the destination of the file)

Now, you can run the clients from any directory in the filesystem without specifying the path to the JAR file as follows:

JConsole

1. $ jconsole
2. Follow the other JConsole steps above.

Java Mission Control

1. $ jmc
2. This will pop up; click on the circled button:

3. This will pop up; click on the circled button:

4. Insert service:jmx:jmxmp://<KubernetesNodeIP>:<KubernetesNodePort>/ in the “JMX service URL” box

(“Connection name” will auto-populate and can be left alone or changed; it doesn’t matter)

5. Click "Finish"

6. In the left-hand panel, double-click the connection name and then click “MBean Server”

7. Metrics and MBean information should now be visible

(J)VisualVM

This method does not seem to work for (J)VisualVM, so the class path still needs to be specified (as detailed above).

Conclusion

If the steps are followed, then the reader will be able to make a remote JMX connection in order to look at the metrics and MBeans information of a given service in Kubernetes.