Automated Grafana Annotations from Flux Events

Grafana annotations are visual markers that allow users to add extra context to dashboard panels. By annotating significant events, users can highlight moments in time where changes to the monitored environment occurred.

In an environment where changes can occur frequently and at any time, annotations provide an effective way to capture what happened and when. By comparing multiple monitored signals around the same annotation, users can accelerate their troubleshooting efforts by correlating the annotated change with observed changes in the dashboard visualizations. This makes annotations a valuable addition to a user’s observability tools.

Using Flux and Kubernetes as an example, this post covers the components involved in automated annotations from Flux events.

Grafana Annotations API

There are several methods for creating annotations, and this approach utilizes the Grafana API. To create annotations using the API, Flux requires a service account with the Editor role and an associated service account token. More information about service accounts and tokens can be found here.

Once you have obtained a token, create a Kubernetes secret with a field named “token”. After creation, the secret should look something like this:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: grafana-token
data:
  token: <base64 token value>

Flux Provider and Alert

Flux generates various types of events that provide users with insights into the details of the running system. With Flux Providers and Flux Alerts, users can configure notifications for specific events to be sent to external systems. In this scenario, we want Flux to notify Grafana whenever it applies new configurations to the Kubernetes cluster.

A Provider is responsible for encoding the event information and sending it to the configured destination. In this case, the destination is the Grafana annotation API endpoint at /api/annotations. The Provider also includes a reference to the authentication secret that contains the service account token required to access the API. More information about providers can be found here.

Here is an example:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: grafana-provider
spec:
  type: grafana
  address: http://grafana.default:3000/api/annotations
  secretRef:
    name: grafana-token

An Alert is responsible for selecting the specific events that the user wants to act on. The eventSeverity and eventSources fields on an alert are used to filter for desired events. More information about alerts can be found here.

In this example, the Kustomization events will provide details about the configurations applied to the cluster.

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: kustomization-alert
spec:
  eventSeverity: info
  eventSources:
  - kind: Kustomization
    name: '*'
    namespace: default
  providerRef:
    name: grafana-provider

Grafana Dashboard

The annotations created in this setup can now be used in any dashboard. To use them, select the built-in Grafana data source while editing a dashboard’s annotation settings. Users can also choose tags to filter the displayed annotations.

With everything set up, users can now view the annotations as vertical lines directly in the dashboard. Hovering over an annotation reveals additional information.

Conclusion

Annotations add valuable context to your dashboards, allowing you quickly get visual information about notable events in the same view as the monitored signals. With the added context, you can accelerate your troubleshooting efforts. Although this post focused on Flux events, users can use the annotations API to automate adding context from other systems to Grafana.

You can also follow a walkthrough of this use case in the repository linked below. It has a devcontainer with all the tools required to complete the walkthrough.

https://github.com/AAkindele/grafana-gitops-annotations