Monitoring with JMX: How to Integrate Tableau Server with InfluxDB

Concurrent VizQL Sessions from JMX in InfluxDB/Grafana

Telegraf is a great tool to collect information from thousands of different sources, but sometimes you need to complete it with other tools due to source limitations. One of these cases when we want to get monitoring and/or performance information from applications using Java Management Extensions API — an API exclusive for Java VMs — where the client must be written in Java too.

This the second part of the Grafana/InfluxDB monitoring series, focusing on collecting JMX metrics from 3rd party applications like Tableau Server. The previous post can be found here.

We have two options: use Telegraf’s Jolokia2 plugin, or use a standalone JMX agent: jmxtrans. Jolokia2 (a JMX-to-REST-API gateway) is a great solution in case we need to monitor local processes, however, to use it for remote monitoring it has to be deployed in a web container (like OSGi, Tomcat, Jboss). For these scenarios using jmxtrans is more straightforward: it can monitor multiple JMX/RMI endpoints and feed the results to time series databases such as InfluxDB.

Monitoring Tableau Server using its JMX API

Tableau Server containers and most parts of their application are written in Java, thus it was a natural decision from the Tableau engineering teams to make their internal, performance metrics available in JMX/RMI. You can get information about current active sessions in each service, cache/hit ratio in the VizQL servers among many other metrics.

Enable JMX in Tableau Server

To enable JMX ports in Tableau Server, use the following tsm commands:

$ tsm configuration set -k service.jmx_enabled -v true
$ tsm pending-changes apply

After the configuration is deployed, you can check the JMX ports for each service:

tsm topology list-ports | grep jmx[^\.]

These ports are dynamically allocated, so it is advised to set them manually to a fixed port as described here.

Now our Server is ready to serve us with JMX metrics.

Understand our Server JMX Domains, MBeans, Attributes and values

Before we set up jmxtrans to collect JMX metrics, we need to understand what are the available attributes in each service. Some people like to use jconsole for this purpose, I personally favor command-line tools like jmxterm.

Let’s pick one of the vizqlserver JMX ports from list-ports ‘s output and connect to it:

$ wget https://github.com/jiaqi/jmxterm/releases/download/v1.0.2/jmxterm-1.0.2-uber.jar
$ java -jar jmxterm-1.0.2-uber.jar
Welcome to JMX terminal. Type "help" for available commands.
$>open localhost:8789
#Connection to localhost:8789 is opened

We are connected to our vizqlserver JMX service, how exciting it is. The first thing we can do is to list all domains using domains command:

$>domains
#following domains are available
Catalina
JMImplementation
Users
com.sun.management
com.tableausoftware.instrumentation
java.lang
java.nio
java.util.logging
org.apache.commons.pool2
tableau.health.jmx

Catalina domain contains tomcat related information, we have a few JVM related domains too, but tableau.health.jmx is the one what we are looking for. Let’s have a closer look, what beans do we have in it:

$>beans -d tableau.health.jmx
#domain = tableau.health.jmx:
tableau.health.jmx:name=searchservice
tableau.health.jmx:name=vizqlservice

We have two beans in the service, one for searchservice and one for vizqlservice. To see what attributes we have inside vizqlservice bean, use the info command:

$>info -b tableau.health.jmx:name=vizqlservice -t a
#mbean = tableau.health.jmx:name=vizqlservice
#class name = com.tableausoftware.health.performancecounter.jmx.JMXMonitoringView
# attributes
  %0   - PerformanceMetrics (javax.management.openmbean.CompositeData, r)

It has one attribute called PerformanceMetrics. We are getting close, the last step is to get the actual values from this bean attribute:

$>get -b tableau.health.jmx:name=vizqlservice PerformanceMetrics
#mbean = tableau.health.jmx:name=vizqlservice:
PerformanceMetrics = {
  ActiveSessions = 0;
  Bootstraps = 0;
  BootstrapsDeferred = 0;
  BootstrapsDeferredThenPerformed = 0;
  DataserverInserters = 0;
  DataserverLockedSessions = 0;
[...]
  VisualModelCacheHits = 0;
  VisualModelCacheMisses = 0;
  VisualModelCachePartialHits = 0;
  WorkbookAttributesParseCacheHits = 0;
  WorkbookAttributesParseCacheMisses = 0;
 };

These metrics are extremely important in case we want to understand how our Server performs. Since these metrics are stored in these processes’ memory, we do not have to go into the Postgres repository and collect these data with costly SQL queries — everything is a matter of milliseconds.

Now we just have to collect some of the important metrics and store them in our InfluxDB.

Sending metrics to InfluxDB with jmxtrans

You can use rpm to install jmxtrans on Centos, otherwise, you can follow the install instruction on the page.

$ sudo rpm -Uvh https://repo1.maven.org/maven2/org/jmxtrans/jmxtrans/271/jmxtrans-271.rpm 

Configuration files are stored under /var/lib/jmxtrans folder. Let’s create a new file called /var/lib/jmxtrans/tableau.json with the following contents (sample config for one vizqlserver and one backgrounder processes):

{
  "servers" : [ {
    "host" : "localhost",
    "port" : "8789",
    "username" : "<tsm_username>",
    "password" : "<tsm_password>",
    "queries" : [ {
      "obj" : "tableau.health.jmx:name=vizqlservice",
      "resultAlias": "vizqlservice",
      "attr" : [ "PerformanceMetrics" ],
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "https://influxdb/",
        "username" : "<username>",
        "password" : "<password>",
        "database" : "monitoring",
        "tags"     : {"host" : "<hostname>", "env":"<clustername>"}
        } ]
    } ],
    "numQueryThreads" : 2
  } ,
 {
    "host" : "localhost",
    "port" : "8092",
    "username" : "<tsm_user>",
    "password" : "<tsm_password>",
    "queries" : [ {
      "obj" : "com.tableausoftware.instrumentation.meters:00=service,01=backgroundJob,name=jobSucceeded,type=meters",
      "resultAlias": "backgrounder",
      "attr" : [ "Count","FifteenMinuteRate","FiveMinuteRate","MeanRate","OneMinuteRate","RateUnit" ],
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "https://influxdb/",
        "username" : "<username>",
        "password" : "<password>",
        "database" : "monitoring",
        "tags"     : {"host" : "<hostname>", "env":"<env>"}
        } ]
    } ],
    "numQueryThreads" : 2
  } ]
}

If all set, we can start our JMX service:

sudo service jmxtrans start

Results in Grafana

If we log in to our grafana instance (the one what we configured in my previous blog post), we can start consuming the results:

We have tons of vizqlservice metrics available for our dashboards

In the next parts, we will build an execd based input plugin for telegraf to collect additional, non-standard metrics (using TSM API in our case), then put everything together in a nice monitoring dashboard.

Questions? Comments?

Did this help? Or something is not quite right? Or you need help to set it up on your end? Just drop a comment below.