Dashboards, dashboards everywhere (as code)

Photo by Luke Chesser on Unsplash

In a fast pacing world where we find ourselves creating a vast amount of services to increase our value outcome, anything which is not automated is penalized for not being fast enough to scale properly. We have automatic quality gates checking our code after each commit, tests that are run automatically after merging a new feature, fast and automated deployments, but what happens with our monitoring? Is it possible to ease and speed up dashboard creation?

When we decide to use Grafana for our dashboards, we will encounter both benefits and drawbacks.

It has:

• A brilliant and easy UI
• A lot of plugins
• Worldwide adoption...

On the other side:

• it’s difficult to update a dashboard with a similar query repeated over more than fifty similar panels
• you need to deal with other team members who overwrite your changes
• and also, it has never been easy to create its dashboards via code (unless you are fine writing JSONs with hundreds of lines.

If we were able to easily “code” our dashboards, usual deployment pipelines can be run and we could enjoy having customized and repeatable graphs everywhere. Besides, we can easily revert any manual change done on them.

This is now fixed with the use of grafonnet.

Jsonnet is a data templating language, basically an extension of JSON. On top, it exists Grafonnet, a jsonnet library available to “code” our Grafana dashboards. It’s not only easy to use the library but also extends its JSON output in case we need to add sections or customize it.

Let’s see some ideas about how we can speed up our dashboard creation. As said before, jsonnet is a templating language, which means that we can use some additions to enrich our JSONs.

i.e.
Having a JSON like:

{
    "title":"'Own the game' is the Adidas target for 2025",
    "description": "Adidas targets to 'Own the game' as its focus is set on 2025"
}

Some expressions are repeating. Let’s make a template so in 2025 it will be easier to update our JSON:

local strategy_name = 'Own the game'
local target_year = 2025
{
  "title" : "'" + strategy_name + "' is the adidas target for " + target_year,
  "description" : "adidas targets to '" + strategy_name + "' as its focus is set on " + target_year
}

The output will be the same as the normal JSON, but we reduced the required changes (two variable values) in case we need to update ours.

Let’s now start by creating a small dashboard to monitor the nodes in our Kubernetes clusters:

local grafana = import 'grafonnet/grafana.libsonnet';
grafana.dashboard.new(
  'demo1',
  description='Summary metrics about containers running on Kubernetes nodes and mainline kubernetes statistics',
  tags=['kubernetes'],
  time_from='now-1h',
).addPanels(
  [
    grafana.singlestat.new(
    'Number of Nodes',
    datasource='prometheus-production',
    height=8,
    sparklineShow=true,
  )
  .addTarget(
    grafana.prometheus.target(
      'sum(kube_node_info{environment="production"})',
    )
  ),
  grafana.singlestat.new(
    'Number of Nodes',
    datasource='prometheus-staging',
    height=8,
    sparklineShow=true,
  ).addTarget(
    grafana.prometheus.target(
      'sum(kube_node_info{environment="staging"})',
    )
  )
])

And we’ll invoke jsonnet + grafonnet to compile our code into a valid JSON for grafana via:

JSONNET_PATH=grafonnet-lib \
  jsonnet demo/demo1.jsonnet > gen/demo1.json

After importing the resulting JSON, we’ll see one shiny dashboard into our grafana instance:

And also the right content when we open it:

Analyzing what we’ve done:

local grafana = import 'grafonnet/grafana.libsonnet';

We can import objects and methods via importing libraries. Grafonnet is one of them, but you can also create your own and import it into your dashboards.

grafana.dashboard.new(

This will set the main settings of the dashboard (name, description, properties, etc)

  'demo1',
  description='Summary metrics about containers running on Kubernetes nodes and mainline kubernetes statistics',
  tags=['kubernetes'],
  time_from='now-1h',
).addPanels(

addPanels is a function of the grafana.dashboard object, which will allow us to add an array of panels to be shown on the dashboard.

  [
    grafana.singlestat.new(

Similar to the grafana.dashboard, we can create a SingleStat panel with some basic settings

     'Number of Nodes',
      datasource='prometheus-production',
      height=8,
      sparklineShow=true,
    )
    .addTarget(

And a target, which will be in our case a Prometheus query

      grafana.prometheus.target(
        'sum(kube_node_info{environment="production"})',
      )
    ),
    grafana.singlestat.new(

And then we add a different panel to the list.

      'Number of Nodes',
      datasource='prometheus-staging',
      height=8,
      sparklineShow=true,
    ).addTarget(
      grafana.prometheus.target(
        'sum(kube_node_info{environment="staging"})',
      )
    )
  ]
)

After seeing how variables work and how basic dashboards are structured, let’s also get some help by defining simple functions to avoid panel creation repetition:

local grafana = import 'grafonnet/grafana.libsonnet';
local getTarget(cluster_environment) =
  grafana.prometheus.target(
    'sum(kube_node_info{environment="' + cluster_environment + '"})'
  );
local getNodeCountStat(cluster_environment) =
  grafana.singlestat.new(
    'Number of Nodes',
    datasource = 'prometheus-' + cluster_environment,
    height = 8,
    sparklineShow = true,
  ).addTarget(
    getTarget(cluster_environment)
  );
grafana.dashboard.new(
  'demo1-functions',
  description='Summary metrics about containers running on Kubernetes nodes and mainline kubernetes statistics',
  tags=['kubernetes'],
  time_from='now-1h',
).addPanels(
  [
    getNodeCountStat('production'),
    getNodeCountStat('staging'),
  ]
)

Basically, we’ll get the same output, but with additional benefits:

• We only need to maintain one function (`getNodeCountStat`)
• All panels share the same settings (like showing the sparkline)
• The definition of the dashboard itself is now smaller and we are just complementing it with helpers with their own responsibility. This way we don’t have JSON sections repeating over and over again across the dashboard, being difficult to locate and maintain.

Finally, about this repetition, we can even need to easily expand this dashboard to an ever-growing list of environments. For this task, we can make use of loops iterating over an array:

local grafana = import 'grafonnet/grafana.libsonnet';
local environments = ['production', 'testing', 'staging', 'development', 'playground'];
local getTarget(cluster_environment) =
  grafana.prometheus.target(
    'sum(kube_node_info{environment="' + cluster_environment + '"})'
  );
local getSingleStat(cluster_environment) =
  grafana.singlestat.new(
    'Number of Nodes',
    datasource = 'prometheus-' + cluster_environment,
    height = 8,
    sparklineShow = true,
  ).addTarget(
    getTarget(cluster_environment)
  );
grafana.dashboard.new(
  'demo1-functions',
  description='Summary metrics about containers running on Kubernetes nodes and mainline kubernetes statistics',
  tags=['kubernetes'],
  time_from='now-1h',
).addPanels(
  [ 
    getSingleStat(environment),
    for environment in environments
  ]
)

As seen, just by extending the array, now we can get a longer list of panels with the required information:

And now all panels can be customized at the same time, if needed, by simply modifying a single helper method.

Finally, it’s time to create a good set of common use case functions and wrap them up into an additional library to be imported by all teams from your organization. This way you’ll make your company rely on the same code base for dashboard creation, speeding up their processes and ensuring they are repeatable, easier to maintain, and faster to deploy.