Making a Dashboard work for you, part one
It’s five o’clock, do you know where your servers are?
Servers and endpoints generate huge amounts of data today. DevOps techniques enable far more servers per OPs employee than legacy systems did in the past. DevOps also creates an ecology where servers and endpoints come and go in a virtual environment, so there’s no walking into the machine room and seeing red LEDs. Managers struggle to digest and present their system data to upper management, and they struggle to comprehend the current system state, a process now called ‘observability.’
Data is not Knowledge, and Knowledge is not Understanding
- Collecting system data is a non-trivial task, even with modern tools.
- Raw data itself is not really comprehensible, due both to volume, and to granularity. It must be filtered in some way.
- Filtered data over time can contain knowledge about your system, but it still isn’t comprehensible.
- Understanding only comes with filtered data and presentation in a human-readable format.
- Alerts are essential, but have to be fine-tuned to hit the sweet spot between ignorance and noisiness.
- Too much data, plus too little understanding of that data, is rampant today.
- Even ancient analog systems have better read-outs than most software systems.
You have multiple systems doing useful things and you’d like to know their status at a glance. But writing your own application is a lot of work, and worse, it creates an ongoing maintenance burden that’s likely too costly for consideration. What’s available out there?
Grafana has a lot of capability:
Grafana is actively maintained and widely used for a variety of visualization chores.
We chose to use Grafana as part of a CI/CD Pipeline project. Our high-ceilinged space came equipped with hanging projectors and roll-down screens. So we decided to dashboard the current status of our Pipeline up in the air above the cubes. Here’s an overview of what we discovered.
Quick Grafana Pros and Cons:
- Free and open source
- Modern components
- Linux, Windows and MacOS support
- Many integrations with data sources
- Comes as Docker container
- Has paid version as SaaS https://grafana.com/cloud/grafana ( if you do this you don’t need to read any further, bye)
- Uses SQLite ( opt. MySQL or PostGres)
- Allows GUI creation of dashboards
- Plugin architecture that’s easy to extend
- Community forum and open features/bug tracking
- Has stateful data stored in the filesystem
- Available GUI dashboard tools often don’t include basic requirements
- SVG graphics aren’t included, nor animation
- Theme system is poorly thought-out
Dashboard basics: Things you need in order to run dashboards…
- Data from applications, sensors, servers, public data stores, and so forth.
Typically this is streaming data like a log file, with timestamps and some sort of metrics. For example: temperature, wind speed, humidity. But it can be single-point data like current water level.
- Data stores of some kind — databases or other ways to collect and organize the incoming stream of data.
These are specialized, but come as plugins to Grafana, so relatively easy.
- An idea of what data metrics are useful and what to ignore.
This is harder than you think. There is too much data by definition, you just want the essentials.
- Graphics tools to display changing data in real time, without refreshing the page.
Grafana does this.
- Webserver to supply the graphics to users.
Grafana does this.
Skills required to setup a well-organized Grafana :
- Setup Grafana
- Sys admin on your chosen platform
- Minor Docker or Apt/Yum experience
- Some understanding of data organization
- Minor graphical skills ( don’t mix your plaids, or wear paisleys
- Connect to data sources
- Sys admin on your data sources
- Understanding of what you want to achieve
- Ability/privileges to connect to other hosts in your system
You can have a Grafana server setup and running in a few minutes, but you’ll need a host, either hardware or an instance in the cloud, and some system admin ability. The Grafana documentation is good, so I won’t duplicate that- http://docs.grafana.org/installation/docker/
You can also install via Apt or Yum, or on Windows or MacOS.
If you want to experiment, run a Docker container on your MacOS or Linux box.
- Grafana is a webserver. It serves dashboards as a webpage, one URL per dashboard. Each dashboard is built out of rectangular components called “panels” which are stacked in a grid. Each dashboard panel has a datasource, which is installed for use by the whole Grafana host as a plugin. There are a small number of included panel-types, and quite a few more that can be installed directly from the Grafana site.
- Dashboard = webpage or URL
- Panel = graphical element where all the fun happens
- Datasource = data collector like a database which can be read by your panels
- Playlists = play dashboards, one by one, at intervals, from a single URL
To make a dashboard:
- Install the Grafana test data datasource plugin, from the datasources tab
- Create a dashboard from the manage dashboards tab
- Pull down the selection of panels and drop one onto the dashboard
- Connect this particular panel with the datasource
- At the top of the panel pull down the list and select edit
Now you can follow the documentation and use the GUI to display data in your panel.
Not so good: the themes are problematic
Grafana themes are color palettes for both the admin pages and the user-facing dashboards. By default Grafana starts with the dark theme, which is the one you’ll see if you Google “grafana images.” It also has a light theme, easily switched. But the theme functions as CSS for both the admin and editor pages and the dashboards. It’s nice but not critical to have a theme for the admin pages, but I really want to control the look of my dashboards, one by one. Furthermore you can’t just edit the CSS for the theme, it has to be baked in at build time, which most of us won’t want to bother with. If you modify the code for panels, they have an individual CSS file, but it’s not easy to get it to override the theme settings. Given the way CSS works, I expected there to be a cascade from app to dashboard to panel. But it’s not done that way. (Angular developers may have no problem with this)
You can take control of the colors and other attributes of your dashboards, by forking and modifying the panel code. I’ll cover this in the second part of this article.