How to productize a Splunk dashboard: Testing
Developed by: Anna Turu Pi, Sanne Rijkers, Ajengui Oumayma, Robbert Zijp and Niels Denissen
TL;DR
VIDI is a data analytics solution build by Vanderlande that provides insights into the performance of our Material Handling Systems. In order to productize these efforts, we’ve started testing our Splunk dashboards. To do so we’re using Docker to create a reproduceable environment and Python to scrape the queries we’d like to test, load the test data in Splunk and isolate this data for each query we test. Finally ApprovalTests is used as a framework to develop and manage these tests.
In this blog post we’ll take you along in our current approach of testing Splunk dashboards for a product we’re developing within Vanderlande, called VIDI. Before we dive in, let’s start with a brief introduction into Vanderlande and the VIDI product.
Vanderlande and VIDI
Vanderlande is the global market leader for future-proof logistic process automation at airports, and in the parcel market. The company is also a leading supplier of process automation solutions for warehouses. With this scale it’s probably no surprise that we are gathering lot’s of data from these Material Handling Systems (given consent of the customer of course). In order to provide our customers with a clear overview of these complex systems, we’ve created VIDI.
VIDI started out as an analytics solution that was build for each customer site individually. This way we were able to quickly validate it’s usefulness and adjust course where needed. Over time we realized that even though each of these customer sites are unique, they share many of the same building blocks and to a certain extend require very similar metrics. We’ve leveraged these commonalities between customer sites to create a single product. This way of working allows us to spend time improving our product for all customers instead of focusing our efforts re-implementing similar logic for each one individually.
But how do we know these dashboards are showing us the correct information and how do we ensure that when altering our dashboards, they’ll continue to do so in the future? Testing!
Testing Splunk Dashboards
Automated testing will sound very familiar to most in the context of Software Engineering. When you write your own code, it’s important to validate it’s correctness by writing unit and integration tests. This way a developer builds confidence that his or her implementation is correct and ensures that when parts of the codebase are altered in the future (potentially by others), this will not break any existing functionality.
These are the guarantees we needed for our product as well and thus we set out to test any query/data transformation/dashboard we create. VIDI is build on top of Splunk and thus we needed the following features:
- A reproduceable Splunk environment
- A way of calling a specific query specified in a Splunk dashboard
- The ability to load specific test data into the platform and isolate it when running a test for a certain query
- A test framework
We’ll discuss each of these challenges and describe how we solved it.
1. Reproduceable Splunk environment
In order to test something, you usually want to adjust the input parameters and validate expectations on the output. Ideally all other variables are fixed. Using Docker we can get very close to recreating the same environment every time we run a test.
Splunk already provides Docker containers so this step was easy to achieve (unfortunately their startup time is quite long though, due to a complete install using Ansible happening for every Splunk container). We’ve setup a framework with our own Dockerfile, docker-compose and Makefile to easily start Splunk and install our app inside the container.
2. Calling a query from a Splunk dashboard
After setting up a reproduceable environment with our Splunk app installed, we needed to find a way to extract the queries that are being run in the dashboards in order to test them individually. To do this we create a small script that parses the Splunk Dashboard XML and returns all queries present in this dashboard.
You’ll find our initial version of such a snippet below:
Of course Splunk dashboards can be much more complex than this script currently captures, but for the time being it suffices for our use-cases.
Finally we use the Splunk Python SDK to execute the queries on the Splunk instance running in Docker.
3. Loading and isolating test data
The queries we’ve extracted above can now be run individually, but we’d like to isolate the data we use for each query as well. This way we can craft a specific dataset to best test a query.
In order to achieve this we follow 2 steps:
1. Load a file with test data for a specific testcase in a Splunk index (e.g. test-data.log)
2. Inject a source filter into the query we want to test (e.g. | search source=test-data.log)
Below you’ll find the code we use to load data into a Splunk index. Splunk will automatically add the source field (name of the file) when ingesting it. You’ll notice we first delete any data in case it was updated, then load the data and finally validate we get the same amount of rows from the index that are present in the original file.
In order to delete data from a Splunk index, we need an extra permission to be granted to the admin user. Below we’ve added the snippet of configuration that we added to the Splunk docker container in order to achieve this. More info on how to do so you can find here.
After loading data into the index, we’ll need to adjust our Splunk queries to filter out this data. For this we’ve written a few functions that enable us to inject a source filter into an existing query. The result of this inject will look something like this:
Before: index="some_index" | stats countAfter: index="some_index" | search source="some_test_file" | stats count
This approach allows us to craft a combination of test data and expected results for each of the queries we specify in our dashboards.
4. Test Framework
To run our testcases from Python we’re using a library called ApprovalTests. This framework enables validating the output data of a query using files instead of in memory objects. Since we’re dealing with the results of queries on output of our tests, often we have quite complex results to validate. By comparing them in the form of files we ease this task.
In a next blog post we’ll elaborate more on the way we use this framework. For now, if you’d like to know more, you can check out the website for ApprovalTests.
We hope this blog has been an interesting read and has inspired you to start testing your dashboards as well. The snippets aim to provide a good picture of how we’ve setup our tests, but if you’ve got questions/suggestions/comments or would simply like to discuss further, feel free to reach out!
About the VIDI team
VIDI stands for “Vanderlande Industries Data Intelligence” and it captures the purpose of our team well. We’re providing Data Intelligence solutions to our external as well as internal customers in the form of dashboarding, Process Mining, Predictive Maintenance and more. We do this with a team of roughly 12 people specializing in data engineering and data science. And we’re hiring ;)
