Akka: monitor your applications with Lightbend Telemetry, Prometheus and Grafana Dashboard

Published in

Akka-Scala

7 min readApr 18, 2022

Me and Dr Mark Grechanik are currently working on a research project with Akka Actor and we are exploring different ways to monitor an application in order to obtain useful insights on the system.

In this article we are going to setup and run Lightbend Telemetry in our Akka application written in Scala using sbt. The process is almost equivalent for Java and any other build tools like Maven and Gradle.

Anytime you build a production application, mainly highly distributed ones, having insights and metrics to track the behavior of that application is critical for your success.

Understanding highly distributed and concurrent systems can be challenging.

Potentially we have a lot of things going on at different times and on other machines. We need somehow to collect insights to understand what is going on in the system and the causes and the effects of our code. These insights can help us debug our application, proactive system maintenance (understanding the causes and the effects), ensure the quality of the service you provide, and much more. You need some way to understand what is happening in your production application. Technically you can write the monitoring functionalities of your system on your own, but this means that you will have to invest a lot of time and a lot of resources trying to wire in the necessary telemetry, and in doing so, you have to scatter that code into your application.

Lightbend Telemetry is a commercial product designed to provide a very non-intrusive way to gather insights from your Akka applications. It exposes various metrics that can then be visualized using multiple tools. Lightbend monitoring tools will provide a lot of functionalities that will make the monitoring process much easier for you just by integrating it into your application as an external service that you can call through APIs.

Create a new Scala project

To start our journey we have to setup a Scala 2 project. The fastest way to do that is to use the “Hello World” templates that sbt. In order to do that, run

sbt new scala/hello-world.g8

Let’s call our template “telemetry”

A template to demonstrate a minimal Scala applicationname [Hello World template]: telemetry

SBT Version

The first thing we have to do to start using Lightbend Telemtry in our Scala 3 project is to make sure that we have a proper version of sbt for our project, so let’s check project/build.properties and make sure that sbt version is equivalent or higher that 1.0.0.

For our purposes we will use sbt version 1.6.2 and your build.properties will look like this

sbt.version=1.6.2

Commercial credentials

Second, we need to get out commercial credentials from the Lightbend website. Lightbend Telemetry is a monitoring tool under license and in order to get the credentials a subscription is needed.

Create a Lightbend Account here

Once you have created your Lightbend Account, retrieve the platform credentials from here

Go under the section credentials and sbt.

Create a file named lightbend.sbt in your project root and copy and paste the credentials you get from Lightbend website. Your file will look like

ThisBuild / resolvers += "lightbend-commercial-mvn" at "https://repo.lightbend.com/pass/mykey/commercial-releases"ThisBuild / resolvers += Resolver.url("lightbend-commercial-ivy", url("https://repo.lightbend.com/pass/mykey/commercial-releases"))(Resolver.ivyStylePatterns)

Pay attention to this known bug

If using sbt version 1.[0–2] you will need to set updateOptions := updateOptions.value.withGigahorse(false) in your build.sbt to avoid a known bug which could cause an issue with resolution.

Add the Cinnamon Plugin to our project

In order to add the Cinnamon plugin, we have to create a file named plugins.sbt under the directory /project. The content of the file will look like this

addSbtPlugin("com.lightbend.cinnamon" % "sbt-cinnamon" % "2.16.2")

Enable Cinnamon in build.sbt

Next step is to enable Cinnamon from our build.sbt file, in order to do that we will add enablePlugins (Cinnamon) to our root project.

Our build.sbt will look like

lazy val root = (project in file(".")).
  settings(
    inThisBuild(List(
      organization := "com.nuklex.scala",
      scalaVersion := "2.13.8"
    )),
    name := "telemetry"
  ) enablePlugins Cinnamon

If using Lightbend Telemetry (Cinnamon) version 2.12.x or older you will need to set enablePlugins(CinnamonAgentOnly) instead of enablePlugins(Cinnamon) (in build.sbt) so that the correct resolvers are used.

Enable Lightbend Telemetry

So far we have setup our project to include the plugin of Lightbend Telemetry. Next step is to enable it, to achieve that we have to explicitly add in our build.sbt

// Add the Cinnamon Agent for run and test
run / cinnamon := true
test / cinnamon := true

Our build.sbt will look like this in the end

lazy val root = (project in file(".")).
  settings(
    inThisBuild(List(
      organization := "com.nuklex.scala",
      scalaVersion := "2.13.8"
    )),
    name := "telemetry"
  ) enablePlugins Cinnamon

// Add the Cinnamon Agent for run and test
run / cinnamon := true
test / cinnamon := true

Run Akka Actor application with Lightbend Telemetry

And now the fun part, let’s add Akka Actor ( “com.typesafe.akka” %% “akka-actor” % “2.6.19”)to our project and the Lightbend Telemetry dependency (Cinnamon.library.cinnamonAkkaTyped) and the related Prometheus dependencies.

lazy val root = (project in file(".")).
  settings(
    inThisBuild(List(
      organization := "com.nuklex.scala",
      scalaVersion := "2.13.8"
    )),
    name := "telemetry",
    libraryDependencies ++=
      Seq(
        // Akka Actor
        "com.typesafe.akka" %% "akka-actor" % "2.6.19",
        // Lightbend Telemetry
        Cinnamon.library.cinnamonAkkaTyped,
        Cinnamon.library.cinnamonPrometheus,
        Cinnamon.library.cinnamonPrometheusHttpServer,
      )
  ) enablePlugins Cinnamon

// Add the Cinnamon Agent for run and test
run / cinnamon := true
test / cinnamon := true

Let’s configure our application.conf to report Lightbend metrics and the Prometheus backend exporters.

For this example we will be reporting actors by class. The config will be automatically fetched from the application.

cinnamon.application = "telemetry"

cinnamon.akka {
  actors = {
    default-by-class {
      includes = "/user/*"
      report-by = class
    }
  }
}
cinnamon.prometheus {
  exporters += http-server
}

Let’s create a HelloWorld Actor to test our Telemetry sandbox.

The class will look like

import akka.actor.Actor

class HelloActor extends Actor {
  def receive = {
    case "hello" => println("hello back at you")
    case _ => println("huh?")
  }
}

Now let’s implement a simple Main to send messages to our Actor

import akka.actor.{ActorSystem, Props}

object Main extends App {
  val actorSystem = ActorSystem("actorSystem")
  val actor = actorSystem.actorOf(Props[HelloActor], "helloActor")
  (1 to 10).foreach { _ =>
    actor ! "hello"
    Thread.sleep(3000)
  }
}

Let’s run the application with sbt via command line and make sure that everything’s working as expected. Run

sbt clean compile

And

sbt run

In the output console you should see

[info] running (fork) Main 
[info] [INFO] [04/17/2022 20:24:52.112] [main-1] [Cinnamon] Agent version 2.16.2
[info] [INFO] [04/17/2022 20:24:52.427] [main-1] [Cinnamon] Agent found Java Futures version: 1.8.0_321
[info] [INFO] [04/17/2022 20:24:52.536] [main-1] [Cinnamon] Agent found Akka Actor version: 2.6.19
[info] [INFO] [04/17/2022 20:24:52.538] [main-1] [Cinnamon] Agent found Scala version: 2.13.8
[info] [INFO] [04/17/2022 20:24:52.577] [main-1] [Cinnamon] Agent found Akka version: 2.6.19
[info] [INFO] [04/17/2022 20:24:52.594] [main-1] [Cinnamon] Agent found Scala Futures version: 2.13.8
[info] hello back at you
[info] hello back at you
...

As you notice sbt will automatically invoke the lightbend agent and start it.

If instead you want to run the application from your IDE you should specify the lightbend jar agent path manually.

For example in IntelliJ you should go to configuration and add

-javaagent path/to/cinnamon-agent.jar

sbt does it automatically.

Download Grafana Monitoring Dashboard and Prometheus Backend

The fastest way to see the insights of Lightbend Telemetry on our machine is via Grafana frontend dashboard, Prometheus backend. In this guide we will use the docker container image of that combo to visualize the insights from our application.

First thing first install docker https://docs.docker.com/get-docker/

Then make sure that docker is up and running and download the docker image of the developer sandbox from here

Unzip the archive and cd into the directory cd cinnamon-prometheus-docker-sandbox-2.16.2/

Now run docker-compose up to start the sandbox and go on your browser at the address http://localhost:3000/login. You will find the Grafana login