Exposing JMX MBeans and Dropwizard metrics in Prometheus format using Akka Http

So recently we started to migrate our infrastructure to Kubernetes. And of course we had to decide how to collect metrics automatically from all services. So we’ve chosen Prometheus. It could be configured to watch Kubernetes Pods and automatically collect metrics via configurable port and endpoint. I didn’t find any ready to go solution, and it turned out that it was really easy to factor my own. So here it is.

First you need to add Akka, Dropwizard and Prometheus Java library to your build.sbt

lazy val akkaHttpVersion = "10.0.5"
lazy val metricsScalaVersion = "3.5.6"
lazy val prometheusVersion = "0.0.26"
libraryDependencies ++= Seq(
// Akka Http
"com.typesafe.akka" %% "akka-http" % akkaHttpVersion,

// Scala API for dropwizard metrics
"nl.grons" %% "metrics-scala" % metricsScalaVersion,

// Provides exporters for HotSpot JVM metrics from MBeans
"io.prometheus" % "simpleclient_hotspot" % prometheusVersion,

// Provides exporter that works as an adapter between Dropwizard and Prometheus registries
"io.prometheus" % "simpleclient_dropwizard" % prometheusVersion,

// Common stuff that would help us with encoding response in Prometheus format.
"io.prometheus" % "simpleclient_common" % prometheusVersion
)

Now let’s define an endpoint

import java.io.StringWriter

import akka.http.scaladsl.model._
import akka.http.scaladsl.server.Directives._
import akka.http.scaladsl.server.Route
import io.prometheus.client.CollectorRegistry
import io.prometheus.client.exporter.common.TextFormat
import scala.collection.JavaConverters._
object PrometheusEndpoint {
private val `text/plain; version=0.0.4; charset=utf-8` = ContentType {
MediaType.customWithFixedCharset(
"text",
"plain",
HttpCharsets.`UTF-8`,
params = Map("version" -> "0.0.4")
)
}

private def renderMetrics(registry: CollectorRegistry, names: Set[String]): String = {
val writer = new StringWriter()
TextFormat.write004(writer, registry.filteredMetricFamilySamples(names.toSet.asJava))
writer.toString
}

def apply(registry: CollectorRegistry): Route =
(get & path("metrics" / "prometheus") & parameter('name.*)) { names =>
val content = renderMetrics(registry, names.toSet)
complete {
HttpResponse(entity = HttpEntity(`text/plain; version=0.0.4; charset=utf-8`, content))
}
}
}

As you can see it pretty straight forward. First we define custom Content-Typefor Prometheus format. Then we say that we will serve metrics at /metrics/prometheus and will take into account name parameters to allow filtering. Then we use aTextFormat helper to render metrics. And complete request processing with response.

Next we need to register some exporters (see comments)

import io.prometheus.client.CollectorRegistry
import io.prometheus.client.dropwizard.DropwizardExports
import io.prometheus.client.hotspot.DefaultExports
import nl.grons.metrics.scala.DefaultInstrumented
import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import akka.http.scaladsl.Http
object App extends App with DefaultInstrumented {

// Add JVM metrics exporters to default registry
DefaultExports.initialize()

// Add Dropwizard adapter exporter
CollectorRegistry.defaultRegistry.register(new DropwizardExports(metricRegistry))

val prometheusEndpoint = PrometheusEndpoint(CollectorRegistry.defaultRegistry)

implicit val system = ActorSystem("system")
implicit val materializer = ActorMaterializer()

Http().bindAndHandle(prometheusEndpoint, "0.0.0.0", 9000)
}

Now don’t forget to add annotation to your Pod template to enable scraping.

annotations:
prometheus.io/scrape: "true"
prometheus.io/path: "/metrics/prometheus"
prometheus.io/port: "9000"

That’s it.