Build your IoT frontend with Grafana
Introduction
As probably all who have used or developed IoT devices, there comes a time when the data has to be show to clients or presented internally to the team. In both cases you need a way to visualize data to transmit you point across to people that may not even be of a technical background. Though they may be many ways to do this, at Worldsensing we have used Grafana to build a wide range of front ends for our data with a great degree of success. Be it an internal tool to monitor on field tests of our hardware to a full fledged frontend that can control our devices remotely and is integrated with an alert system.
Data visualization in 10 minutes
In comparison to other tools we have used in the past for visualization, I want to highlight the degree of simplicity and little time required to get you data on screen. To do so we will show how setup the visualization of your data in 10 minutes.
Prerequisites
- A Linux PC (preferably Ubuntu 16.04)
- Zip file containing example data and scripts
Unpack the file and deploy the Grafana+InfluxDB stack:
tar -xvzf basicinflux_demo.tar.gz
cd basicinflux/ && sudo ./deploy.sRun the inserter script located in the same directory:
python3 ./inserter.py -i ./ls_data_4145.txtThis script reads Loadsensing datalogger data from a file, but it could be easily modified to read the data from a Loadsensing LoRa gateway via an HTTP post. The main point of this script is to serve as an example that the reader can tailor to his needs.
Log into Grafana interface with a web browser and setup the InfluxDB datasource as show in the image below:
Configuration > Datasources > Add datasource
Proceed to create a new dashboard:
Create > dashboard
Here we will also create two dashboards variables that we will use later.
Dashboard settings > Variables > Add variable
We will configure it as follows:
We do the same thing and configure the second variable as follows:
We will start by showing the raw data as it arrives and make sure the inserter script is working.
Add a Graph panel to the dashboard and configure it as follows:
As show we will select only the channel 0 for now and set the auto-refresh to 5s.
Suppose we are testing the stability of our data, then we might also want to show the average value over the shown period and the standard deviation of our data. That can be done adding two Single Stat panels and configure them as following:
The resulting dashboard will be similar to this.
Now, when the data has to be shared with your team, you rarely show the raw data. More often than not you want to process the data by applying some filter to make more sense of it. We will apply a pretty common smooth filter to our data. To do that edit the graph panel and change its configuration as follows.
We will also tweak the legend display configuration for this graph.
And we end up with the following dashboard.
To see the data of the other channels we could simply change the Channel value from the drop down menu in the top, but we could also create a row for each channel in the dashboard. To do that insert a new Row from the new panel menu and configure it as shown here.
Finally we can select the All option from the channel drop down menu and the dashboard will generate the same visualization elements for all channels.
You can apply other transformations as desired and play with the different elements to see how to better present the data. The limit is the expressiveness of the Influx query language which is reasonably powerful, and a detailed description of the functions it offers can be found here.
Grafana as an IoT Frontend
In this second part I will to highlight some of the Grafana components and features used in our R&D project frontend. This frontend consists of several dashboards linked among them each presenting a different view of our sensors’ data and the monitoring information of each our our devices. The data and processing backend is provided by InfluxDB, Kapacitor and a simple backend software that provides an API that glues all the systems together.
This frontend integrates several models of Worldsensing Loadsensing dataloggers as well as jammer detector devices that monitor LoRa and WiFi frequency bands.
Among other features it might be worth highlighting the following:
- Security information and event management (SIEM) integration
- Alerts on settable thresholds for each sensor
- Integrated remote control of the dataloggers
Graphs
Graphs and stats panels are the bread and butter of a data visualization dashboard and as such there is plenty information on the internet to make a good use of them.
Nevertheless, I would like to bring attention to the great potential that can bring using these standard components in conjunction with a custom API to feed it data. In Grafana you are not limited to query a database as your sole method to get data to plot, There is a SimpleJson datasource plugin that allows you to get your data from using HTTP queries to an API.
In our frontend we use this method when a direct InfluxDB query would be too costly or it simply cannot be done due to data needed from 3rd parties. For instance our backend software can load all the necessary data and use any powerful library, like python’s pandas, to process the data before sending it to Grafana to be plotted.
The text panel
The text panel is one of the most basic panels in Grafana, usually just used to tag some parts of the dashboard, put some description and any other static content.
As you might have noticed in the image above the Text panel accepts inputs in markdown, but also in HTML, which means you can insert any valid HTML tag into it and it will be rendered as part of the dashboard. It also means that you can embed javascripts into it and what is more, Grafana templating works within all the HTML code you put into it.
The end result is that we can implement some simple control objects into a Grafana Text panel, In the example to the right we have a form that allows us to set the alert thresholds for each of our sensors, which will trigger notifications on 3rd party systems as well as a visual feedback.
As seen in the code below it is basically a simple ajax call that interacts with our backend API, which does most of the heavy lifting. However, this means that we do not need an external configuration page or other frontends, the controls are smoothly integrated into our dashboards, and thanks to the power of templating, a new one will be generated for each datalogger that is installed.
The Ajax panel
Cranking it up one more gear, we have the Ajax panel developed by a 3rd party (v0.0.3). This component will basically display whatever is the result of an HTTP call to the given URL. As a bonus it will update the content of the panel at the rate the dashboard is set to refresh.
The use we do of this component is in many ways similar to the Text panel with the added advantage that the HTML code is not embedded into the panel but it is dynamically generated by our backend API.
In the image to the right the panel will display the generated controls to set the sampling rate for each of our dataloggers. In this case the backend will take into account the type of sensor, the recent data and current configuration of that datalogger to show us the available options and what the current configuration is.
The configuration of this panel is simply our API endpoint that will serve the content.
Conclusions
We have seen only a small subsets of the dashboard components that Grafana has to offer, yet only with those we can easily build a pretty full featured frontend for any application we need. We are yet to talk about the map panels, the annotations features of the graphs or the user management and multi-tenancy built in, but that may be in the future. For now it will suffice to say that building a frontend for an IoT system in Grafana is easy.
