Remotely connecting through Kubernetes to a JMX agent in a Spring Boot 1.x application to monitor an application

Tremaine Eto
Oct 31, 2018 · 6 min read

What‘s JMX, Kubernetes, and Spring Boot?

A resource is instrumented with Managed Beans (MBeans) that are registered in a core-managed MBean server which can run on devices that have the Java programming language installed.

Kubernetes is an open-source container-based system that automates deployments, scaling, and general container management. In a use case relevant to this article, it can orchestrate Java applications that are deployed in Docker containers.

Spring Boot is a Java framework for running a Spring application to support stand-alone, production-grade applications. It includes features such as built-in annotations and embedded HTTP servers like Tomcat or Jetty.

What’s the problem?

However, once a Spring Boot application is deployed in Kubernetes, the connection somehow ceases to work, at least in my case when I tried doing so.

After searching for possible solutions online, it became clear that this issue was somewhat common, as evidenced by multiple threads on Stack Overflow. Unfortunately, none of the solutions provided seemed to directly work for me, and none of them seemed directly tailored to Kubernetes specifically.

A large part of the issue is that in a remote connection through a typical RMI connector, then a second random port is essentially chosen for a user’s application as an accessible JMX endpoint. It’s essentially impossible for someone to dynamically be able to guess this endpoint. This blog post by Mark Feeney goes so far as to say “(Don’t Use) RMI” — instead, he (among others) suggests using the JMX Messaging Protocol, or JMXMP.

This article will just step through the process I went through for getting a remote JMX connection to work through Kubernetes via JMXMP; it is likely not the only possible solution to the issue but hopefully will add to the overall solution set.

Changes you need to make in your Spring Boot application

The first of which is to create a class that exposes a new @Bean; this bean will have to be a ConnectorServerFactoryBean, and it essentially needs to have its service URL set to a dynamic JMX URL that is forced onto port 1099 instead of a random port. A code example is as follows.

@Bean
public ConnectorServerFactoryBean connectorServerFactoryBean() throws Exception {
final ConnectorServerFactoryBean connectorServerFactoryBean = new ConnectorServerFactoryBean();
String jmxURL = "service:jmx:jmxmp://localhost:1099/";
connectorServerFactoryBean.setServiceUrl(jmxURL);
}

As you can see, the server URL that you would be able to access the JMX agent is now through the JMXMP protocol: service:jmx:jmxmp://localhost:1099/.

Additionally, ensure that the following property is set in the application.properties (or application.yml) file of your Spring Boot application:

spring.jmx.enabled=true

Now, deploy your Spring Boot application to Kubernetes through your usual process.

Changes you need to make in Kubernetes

Ultimately, you will access the JMX agent through this NodePort as the port and through the pod’s NodeIP as the IP.

How to use the JMX clients to connect

When using any JMX client (such as JConsole, Java Mission Control, VisualVM, etc.), I’ve found that you must specify the location of your opendmk_jmxremote_optional_jar JAR file as an argument or by adding it to your Java installation’s “endorsed” directory. Below are the steps to do each method.

Method 1: Specifying endorsed directory with JMX client

JConsole

  1. $ jconsole ‘-J-Djava.endorsed.dirs=.’
  2. You will see this pop up (if not, you may need to install JConsole):

4. Under “Remote Process”, insert the following: service:jmx:jmxmp://<KubernetesNodeIP>:<KubernetesNodePort>/

5. A successful connection should be made and the necessary MBeans and metrics should be viewable

(J)VisualVM

2. $ (j)visualvm (dash dash) cp:a path/to/your/opendmk_jmxremote_optional_jar

Note: you can add the “j” if you want to use JVisualVM. Also, replace the (dash dash) with two hyphens; Medium automatically converts this to an em dash, which is not what you want to use.

3. This will pop up; click the button circled:

4. In the “Connection” box, enter service:jmx:jmxmp://<KubernetesNodeIP>:<KubernetesNodePort>/

5. Click “Do not require SSL connection”

6. Click OK

7. Under “Remote”, you will now see the KubernetesNodeIP show up; you’ll be able to doubleclick on the service URL and see the metrics and MBean information

Method 2: Setting up local Java endorsed directory

  1. $ cd path/to/your/jdk1.x.x_xx/lib
    ( To find your machine’s JDK installation path, you can run $whereis java and then $ cd /lib)
  2. $ mkdir endorsed
  3. $ cd endorsed
  4. $ cp path/to/your/opendmk_jmxremote_optional_jar-1.0-b01-ea.jar .
    (Don’t forget the period at the end! This specifies the current directory as the destination of the file)

Now, you can run the clients from any directory in the filesystem without specifying the path to the JAR file as follows:

JConsole

  1. Follow the other JConsole steps above.

Java Mission Control

  1. This will pop up; click on the circled button:

3. This will pop up; click on the circled button:

4. Insert service:jmx:jmxmp://<KubernetesNodeIP>:<KubernetesNodePort>/ in the “JMX service URL” box

(“Connection name” will auto-populate and can be left alone or changed; it doesn’t matter)

5. Click "Finish"

6. In the left-hand panel, double-click the connection name and then click “MBean Server”

7. Metrics and MBean information should now be visible

(J)VisualVM

Conclusion

cloud native: the gathering

A gathering place for all things cloud native, microservices, api development

Tremaine Eto

Written by

Software engineer.

cloud native: the gathering

A gathering place for all things cloud native, microservices, api development