Better akka, Spray metrics with kamon

getting inside akka and spray

Kamon is a monitoring tool for the JVM; it works by inspecting/ weaving your code and profiling it in real time. What makes Kamon different from visualVM and other JVM tools is that it’s great for akka. Other JVM tools give us an in-depth analysis of threads running on the VM; they are not great for monitoring an event-driven and asynchronous system such as akka.

At this level, we want an in-depth analysis of the messages being processed by a thread; akka does not package a monitoring extension; Kamon to the rescue :D

How Kamon works

Kamon works by using aspectJ and a trace context. It uses aspectJ to weave the code running; it hooks into the method calls made by akka’s actorSystem; through this hook kamon is able to track and collect real-time metrics on the akka system. The big benefit with this instrumentation is that it require’s little code change in our application.

We are now able to trace our application, the goal being to collect local metrics per actor, and message flow between actors. This gives us a clearer picture of what is really going on in the akka system. With Kamon we can also monitor the JVM and system metrics.

But, why should we monitor?

1. Know the overall health of our application
2. Check code safety
3. Check the applications resilience
4. Know the applications performance

As for akka monitoring we’re interested in finding;

1. Actor throughput
2. Actor latency
3. # of messages processed
4. Avg mail-box size per actor
5. Processed messages per actor
6. Avg time spent message is spent in mailbox — how fast is the actor working

Getting started with Kamon

Update sbt and add kamon dependencies.

"io.kamon"    %% "kamon-core"           % kamonVersion,
"io.kamon"    %% "kamon-annotation"     % kamonVersion,
"io.kamon"    %% "kamon-akka"           % kamonVersion,
"io.kamon"    %% "kamon-statsd"         % kamonVersion,
"io.kamon"    %% "kamon-spray"          % kamonVersion,
"io.kamon"    %% "kamon-log-reporter"   % kamonVersion,
"io.kamon"    %% "kamon-system-metrics" % kamonVersion,
"org.aspectj" %  "aspectjweaver"        % "1.8.1",

Full Build.scala here. We add an sbt dependency (plugins.sbt) next to allow us to load aspectJ agent when we call run. It’s also important to enable the log reporter to log every metric being sent to statsD (graphite). Both kamon-akka and kamon-spray need aspectJ dependency to work; the other modules use tracing, this brings bytecode instrumentation to collect metrics and perform TraceContext propagation on akka.

addSbtPlugin("com.typesafe.sbt" % "sbt-aspectj" % "0.10.0")

Next we update our application.conf and add a kamon key.

Next, we have to start Kamon in order to publish metrics by calling it’s start method.

import kamon.Kamon

Kamon.start

On the kamon key, we add a statsD backend, where we will publish our stats; and the interval we would want to track. Full application.conf here.

When assembling with sbt; you will have to build a merge strategy for the conflicts. There’s a sample aop merge file here.

Setting up the statsD and Grafana backend

Kamon-statsd uses statsD and a graphite backend to publishs stats; this is done over UDP; with the obvious benefit of being very fast. We use Grafana to visualize this metrics kamon publishes. Thanks to the guys at kamon we have a docker image we can install, this image packages statsD, graphite and Grafana and allows us to easily setup.

The ideal monitoring setup

Drawbacks

Kamon is great; monkey patching is expensive though, and certain parts if not off, will degrade your performance drastically. Kamon is a young project, but it’s growing.

“Beware that taking stack traces on every use of the ask pattern doesn’t come for free. When using the heavyweight mode Kamon creates a new Exception and stores its stack trace during the lifetime of the ask, and that will consume some additional CPU and memory.” — Kamon docs