Near real-time monitoring charts with Spring Boot Actuator, Jolokia and Grafana
Spring Boot Actuator gives a nice out-of-box way to watch for applications performance and responsiveness, by /metrics endpoints.
Therefore, in a clustered and elastic environment, applications nodes can grow, multiply and be composed by a very large number of instances. Monitoring isolated nodes can be painful and inconclusive. A tool for aggregating time-series data fits better.
This post is intended to find out a solution for monitoring Spring Boot Metrics over time, in a time-series fashion, with no implementation needed, only by tooling and configuration.
APMs like NewRelic, AppDynamics or DataDog does a great job by using JVM and bytecodes instrumentation to generate their own metrics, insights and relevant transactions. It’s possible yet to annotate methods with @Timed. Therefore, all available resources under Actuator library are ignored. Another issue by using this kind of took is related to data retention, that can obfuscate small time-window data.
spring-boot-admin might be another option, since it connects to Spring Boot instances, aggregate nodes and so on. However, /metrics endpoints are not monitored along timeline, neither aggregated among different nodes of the same application module (horizontal scaling). It means you have both situations: no data-series metrics but only instant and isolated snapshots of Actuator data of isolated nodes.
jconsole and visualvm might be another option, connecting directly to JMX, through RMI, to nodes. Actuator store Metrics data inside a MBean from JMX. Additionally, by using Jolokia, MBeans are exposed to a RESTful HTTP enpoint, /jolokia. So, the same information can be acquired in both endpoints: JMX MBean Metrics and Rest HTTP Jolokia endpoint. Nevertheless, the same problem resides in connecting directly to individual nodes in a clustered environment, additionally with painful RMI old-school protocol. Workarounds ahead.
Moving ahead, I’ve checked if it is possible to solve this issue with some Ops-team modern tools:
- Prometheus: written at SoundCloud, it stores series of monitoring data with nice charts. Prometheus Gauges and Actuator Metrics gauges are not fully compatible, and people written an adapter to convert data. You can also configure Prometheus to collect JMX data.
- Sensu: as a Nagios or Zabbix modern replacement, has a plugin to connect to Spring Boot, but it’s repository hasn’t changed recently, so I decided to give up on this.
- StatsD: Spring Boot has a documentation about custom exporting data to StatsD. However, you have to implement some stubs to get it working, besides install an instance of StatsD along Spring Boot application module.
- Graphite: You got to be a hero to install and get Graphite running. If you get there, you can configure it along StatsD to get metrics working in a chart.
- OpenTSDB: Spring Boot has a documentation about connecting data to OpenTSBD. However, in the same way as StatsD, you have to implement and maintain custom code to make it work. Additionally, OpenTSDB has no chart visualization out-of-box.
- JMXTrans: can be used to extract data and send to other monitoring tools. Implementation needed too.
- Ganglia: Also based on instrumentation on JVM, leaving behind all Actuator resources. Same APM problem.
After some research, I ran into a better solution with InfluxDB and Telegraf and zero coding, only by configuring correct parts.
- Jolokia: Spring Boot endorses using Jolokia to export JMX data over HTTP. You should just add some managed dependencies on classpath, and everything works out-of-box. No implementation needed.
- Telegraf: Telegraf endorses JMX data retrieval through Jolokia integration. It has a pre-made input plugin, which works out-of-box. No implementation needed. Just configuration.
- InfluxDB: InfluxDB receives metrics from Telegraf via output plugin, which works out-of-box. No implementation needed.
- Grafana: Grafana renders charts by connecting to InfluxDB as a data source. It works out-of-box, and as well as other parts of solution, no implementation needed.
In a nutshell, configuring all those parts are quite simple:
That’s it. It’s all working. I’ve put an example configuration in my github project with docker-compose. Check it out!
You can reach me if you have any trouble configuring those monitoring tools.
