Oh Meta….

Revealing Metabase: How Circles.Life provide flawless data insights

The story of Circles.Life started in 2016 when we disrupted the telco industry in Singapore, but its journey doesn’t stop there. Circles.Life is destined to go beyond telco, providing more personalized digital lifestyle services — utilizing on its telco data insights.

This blog introduces Metabase — open source business intelligence tool. It lets you ask questions about your data and displays answers in beautiful formats that make sense, like bar-graph, lines, pie-chart, etc. However, have you ever thought that this tool can turn into a nightmare due to its memory issues and can hang upon loading your insights?

Yes, you read it right. That’s why we, Circles.Life Data team, monitor Metabase very closely. We set up the monitoring mechanism using an open-source stack which consists of :

• JMXtrans — allows you to connect to any number of Java Virtual Machines (JVMs) and query them for their attributes without writing a single line of Java code. The attributes are exported from the JVM via Java Management Extensions (JMX).
• InfluxDB — time series database built from the ground up to handle high write and query loads. It is meant to be used as a backing store for any use case involving large amounts of time-stamped data, including DevOps monitoring, application metrics, IoT sensor data, and real-time analytics.
• Grafana — analytics platform which allows you to query, visualize, alert on and understand your metrics no matter where they are stored. You can create, explore, and share dashboards with your team and foster a data-driven culture.

Under the hood

We use Metabase at its core to monitor various reports that drive business decisions and during the process, we have run into different challenges. Hence, we decided to monitor Metabase at the application level using JMX — a Java technology that manages and monitors applications. This monitoring resource is represented by objects called MBeans, which we refer as JMX metrics.

Metabase exposes JMX metrics and it helps us to setup alerts on its various usage parameters like CPU usage, memory usage, thread counts, process load, etc.

Having said that, let me take you through the set-up process for this monitoring:

Architecture of Metabase monitoring setup

Step 1: Enable JMX for Metabase process

First, we need to specify some system properties that lets the JVM know that we want to allow remote monitoring to get connected to JMX port from remote machine. JVM is a virtual machine that enables a computer to run Java programs as well as programs written in other languages that are compiled in Java bytecode. Assuming we are running Metabase using metabase jar file, we would need to change the metabase process invocation to the below command, which includes these properties:

java -Dcom.sun.management.jmxremote \
 -Dcom.sun.management.jmxremote.port={JMX_Port} \
 -Dcom.sun.management.jmxremote.rmi.port={JMX_Port} \
 -Dcom.sun.management.jmxremote.authenticate=false \
 -Dcom.sun.management.jmxremote.ssl=false \
 -Dcom.sun.management.jmxremote.local.only=false \
 -Djava.rmi.server.hostname={instance_IP} \
 -jar {path_to_metabase.jar}

Now you can use JMXTerm to connect with metabase JVM and see the MBeans emitted. JMXTerm is an interactive command line JMX client.

Step 2: Capture JMX metrics and send to InfluxDB

Next, we need to capture these JMX metrics, like CPU, Memory, etc., and send them to InfluxDB. For that, we use JMXTrans, which has a variety of sinks (writers) for storing metrics at desired location.We also use InfluxDBWriter for our JMX metrics storing requirement in InfluxDB. You can install JMXTrans by following these steps.

Below is the JSON configuration file with which we need to start the JMXTrans process:

{
 "servers": [{
  "port": "{JMX_Port}",
  "host": "{instance_IP_to_capture_JMX_metrics}",
  "queries": [{
   "obj": "java.lang:type=Memory",
   "attr": ["HeapMemoryUsage", "NonHeapMemoryUsage"],
   "resultAlias": "jvmMemory",
   "outputWriters": [{
    "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
    "url": "http://{influxDB_host}:{influxDB_port}/",
    "username": "{influxDB_username}",
    "password": "{influxDB_password}",
    "database": "{influxDB_database}",
    "tags": {
     "application": "{your_application_name}"
    }
   }]
  }, {
   "obj": "java.lang:type=Threading",
   "attr": ["ThreadCount", "TotalStartedThreadCount", "PeakThreadCount"],
   "resultAlias": "jvmThreads",
   "outputWriters": [{
    "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
    "url": "http://{influxDB_host}:{influxDB_port}/",
    "username": "{influxDB_username}",
    "password": "{influxDB_password}",
    "database": "{influxDB_database}",
    "tags": {
     "application": "{your_application_name}"
    }
   }]
  }, {
   "obj": "java.lang:type=Runtime",
   "attr": ["Uptime"],
   "resultAlias": "runtime",
   "outputWriters": [{
    "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
    "url": "http://{influxDB_host}:{influxDB_port}/",
    "username": "{influxDB_username}",
    "password": "{influxDB_password}",
    "database": "{influxDB_database}",
    "tags": {
     "application": "{your_application_name}"
    }
   }]
  }, {
   "obj": "java.nio:name=direct,type=BufferPool",
   "attr": ["Count", "MemoryUsed"],
   "resultAlias": "bufferpool",
   "outputWriters": [{
    "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
    "url": "http://{influxDB_host}:{influxDB_port}/",
    "username": "{influxDB_username}",
    "password": "{influxDB_password}",
    "database": "{influxDB_database}",
    "tags": {
     "application": "{your_application_name}"
    }
   }]
  }, {
   "obj": "java.lang:type=OperatingSystem",
   "attr": ["ProcessCpuLoad", "SystemCpuLoad", "ProcessCpuTime", "FreeSwapSpaceSize", "TotalSwapSpaceSize", "OpenFileDescriptorCount", "SystemLoadAverage"],
   "resultAlias": "operatingSystem",
   "outputWriters": [{
    "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
    "url": "http://{influxDB_host}:{influxDB_port}/",
    "username": "{influxDB_username}",
    "password": "{influxDB_password}",
    "database": "{influxDB_database}",
    "tags": {
     "application": "{your_application_name}"
    }
   }]
  }]
 }]
}

Now start JMXTrans with command:
./{jmxtrans_installation_path}/bin/jmxtrans.sh start {path_to_above_json_file}

Step 3: Create Insights on InfluxDB data using Grafana

Now you can login into your InfluxDB and can see the incoming metrics. We empower these metrics insights by setting up Grafana over InfluxDB. We can also setup alerts on these metrics for crossing a certain level and wire up these alerts to various channels, like Slack. Below is an example of metabase insights dashboard:

Sample Metabase insights on Grafana dashboard

See, it’s quite easy right?

Impact

Our stakeholders gain insights from Metabase to help them with their-day-to-day decisions. These alerts help us prevent Metabase to choke and allow us to restart the JVM process when required.This monitoring exercise prevents Metabase to interrupt the process of insights gathering needed for business reports, and creates a smooth data browsing experience across organizations.

We are on a mission to create a highly personalized digital lifestyle service. To succeed in this mission, we are looking for heroes who want to join on our journey in solving challenging problems across complete engineering paradigm. If you are up for the challenge, apply here to be considered!