From events to Grafana annotation

** This blog has moved! Please read this article here:
https://engineering.contentsquare.com/2019/from-events-to-grafana-annotation/ **

At some point you are frustrated in monitoring. You have to correlate facts to metrics, and it can become really difficult with monitoring systems. You have dozens of metrics and dashboards but finding a cause to a spike is often a needle in a haystack.

The annotation API at a glance

Grafana comes with a native annotation api, that can store and retrieve annotations for dashboards. Adding annotation is very easy and convenient since Grafana 4.6. Just Hold CMD + Click, this will add an annotation at the selected timestamp that will be stored in Grafana database. But, this is clearly a manual step that could be used to explain events afterwards.

Annotations can also be retrieved from other data sources such as ElasticSearch, Mysql/Postgresql, prometheus, but let’s focus on the annotation Api ans its in two formats, the native grafana format and the graphite format.

The Graphite Format

Quick takeaways on choosing the graphite format:

• dashboard has to be configured to display annotations
• four simple fields “what”, “when”, “data”, and “tags ”

Grafana Graphite annotation format

The Grafana native Format:

Quick takeaways on choosing the native format:

• can be linked to a dashboard and panel id (optional).
• can span a time range (with timeEnd)
• millisec epoch

Grafana native annotation format

Manual annotation posting

Posting annotation is fairly easy, just call the right endpoint with the right payload. We’ll use here the graphite API as this annotation is really simple to both implement and understand.

We must be authenticated in order to use the grafana APIs. First create a token as explained here. Once done, here you go

Once posted, the annotation appears on the dashboard

Should I really curl grafana everytime ?

Of course no, there is plenty of way to achieve this. At ContentSquare Engineering we often build simple tool to achieve simple actions. https://github.com/ContentSquare/grafana-annotation is one example. We deploy this tool via ansible on our systems, and can refer to it when needed.

A simple use case is to track Systemd services events as follows:

• step one, deploy the Grafana annotation tool
• step two, create a notifier Systemd service
• step three, update the target Systemd service to track

The Notifier Systemd Service:

The Target Systemd Service to monitor:

This service will trigger the notifier when started, stopped or failed. OnFailure=notify-grafana-failure@%i.service %i refers to the failing service name. See systemd specifiers for more information.

Grafana annotations on dashboard:

By adding the corrects tags to filter annotations, we will display on the whole dashboard all the corresponding annotations.

Grafana annotations options for a dashboard

We now have some clues of what might have happened

To go further:

There are plenty of use cases, but let’s imagine a world where all sort of events could be traced to make sense at some point. When you start a ansible-playbook you could in your ci add a notify step to keep track of actions, deployments…

• the grafana notification API here
• the grafana annotation http_api here