Published in


Akka: monitor your applications with Lightbend Telemetry, Prometheus and Grafana Dashboard

Me and Dr Mark Grechanik are currently working on a research project with Akka Actor and we are exploring different ways to monitor an application in order to obtain useful insights on the system.

In this article we are going to setup and run Lightbend Telemetry in our Akka application written in Scala using sbt. The process is almost equivalent for Java and any other build tools like Maven and Gradle.

Anytime you build a production application, mainly highly distributed ones, having insights and metrics to track the behavior of that application is critical for your success.

Understanding highly distributed and concurrent systems can be challenging.

Potentially we have a lot of things going on at different times and on other machines. We need somehow to collect insights to understand what is going on in the system and the causes and the effects of our code. These insights can help us debug our application, proactive system maintenance (understanding the causes and the effects), ensure the quality of the service you provide, and much more. You need some way to understand what is happening in your production application. Technically you can write the monitoring functionalities of your system on your own, but this means that you will have to invest a lot of time and a lot of resources trying to wire in the necessary telemetry, and in doing so, you have to scatter that code into your application.

Lightbend Telemetry is a commercial product designed to provide a very non-intrusive way to gather insights from your Akka applications. It exposes various metrics that can then be visualized using multiple tools. Lightbend monitoring tools will provide a lot of functionalities that will make the monitoring process much easier for you just by integrating it into your application as an external service that you can call through APIs.

Create a new Scala project

To start our journey we have to setup a Scala 2 project. The fastest way to do that is to use the “Hello World” templates that sbt. In order to do that, run

Let’s call our template “telemetry”

SBT Version

The first thing we have to do to start using Lightbend Telemtry in our Scala 3 project is to make sure that we have a proper version of sbt for our project, so let’s check project/build.properties and make sure that sbt version is equivalent or higher that 1.0.0.

For our purposes we will use sbt version 1.6.2 and your build.properties will look like this

Commercial credentials

Second, we need to get out commercial credentials from the Lightbend website. Lightbend Telemetry is a monitoring tool under license and in order to get the credentials a subscription is needed.

Create a Lightbend Account here

Once you have created your Lightbend Account, retrieve the platform credentials from here

Go under the section credentials and sbt.

Create a file named lightbend.sbt in your project root and copy and paste the credentials you get from Lightbend website. Your file will look like

Pay attention to this known bug

If using sbt version 1.[0–2] you will need to set updateOptions := updateOptions.value.withGigahorse(false) in your build.sbt to avoid a known bug which could cause an issue with resolution.

Add the Cinnamon Plugin to our project

In order to add the Cinnamon plugin, we have to create a file named plugins.sbt under the directory /project. The content of the file will look like this

Enable Cinnamon in build.sbt

Next step is to enable Cinnamon from our build.sbt file, in order to do that we will add enablePlugins (Cinnamon) to our root project.

Our build.sbt will look like

If using Lightbend Telemetry (Cinnamon) version 2.12.x or older you will need to set enablePlugins(CinnamonAgentOnly) instead of enablePlugins(Cinnamon) (in build.sbt) so that the correct resolvers are used.

Enable Lightbend Telemetry

So far we have setup our project to include the plugin of Lightbend Telemetry. Next step is to enable it, to achieve that we have to explicitly add in our build.sbt

Our build.sbt will look like this in the end

Run Akka Actor application with Lightbend Telemetry

And now the fun part, let’s add Akka Actor ( “com.typesafe.akka” %% “akka-actor” % “2.6.19”)to our project and the Lightbend Telemetry dependency (Cinnamon.library.cinnamonAkkaTyped) and the related Prometheus dependencies.

Let’s configure our application.conf to report Lightbend metrics and the Prometheus backend exporters.

For this example we will be reporting actors by class. The config will be automatically fetched from the application.

Let’s create a HelloWorld Actor to test our Telemetry sandbox.

The class will look like

Now let’s implement a simple Main to send messages to our Actor

Let’s run the application with sbt via command line and make sure that everything’s working as expected. Run

sbt clean compile


sbt run

In the output console you should see

As you notice sbt will automatically invoke the lightbend agent and start it.

If instead you want to run the application from your IDE you should specify the lightbend jar agent path manually.

For example in IntelliJ you should go to configuration and add

-javaagent path/to/cinnamon-agent.jar

sbt does it automatically.

Download Grafana Monitoring Dashboard and Prometheus Backend

The fastest way to see the insights of Lightbend Telemetry on our machine is via Grafana frontend dashboard, Prometheus backend. In this guide we will use the docker container image of that combo to visualize the insights from our application.

First thing first install docker https://docs.docker.com/get-docker/

Then make sure that docker is up and running and download the docker image of the developer sandbox from here

Unzip the archive and cd into the directory cd cinnamon-prometheus-docker-sandbox-2.16.2/

Now run docker-compose up to start the sandbox and go on your browser at the address http://localhost:3000/login. You will find the Grafana login

Sign in with “admin” and “admin” you will see the starting dashboard

Enable the Cinnamon Prometheus plugin

Once enabled go to the Akka Actor dashboard

Let’s set “refreshing every” to 5 seconds in the dashboard settings

Now let’s run our simple application via sbt

You should be able to see the insights displaying on the Grafana dashboard.

That’s all!



Akka is a toolkit for building highly concurrent, distributed, and resilient message-driven applications for Java and Scala

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store