Speeding Up Your Splunk Dashboards

When loading up our Splunk Dashboard there may be times that we see a slight delay in our graphs or visualisation loading. When our data grows, this delay could become even longer. Unfortunately, some end users, especially nontechnical users, will not really care about the technology and work going on in the background to get this data delivered and will be a little frustrated with the delay. But fortunately, Splunk provides different ways for us as developers to speed up the way we deliver our data to our dashboard.

Limiting Search Time

This is probably part of Splunk dashboard creation 101, as it is something that you should be learning very early on in the process, but still gets missed. It generally happens when a dashboard is set up when the environment is initially set up with a much smaller data sample. The sample grows and no one thinks to change the dashboard search. A lot of the time, if we are using “All time” for our search, we generally don’t need all of that data, so your dashboard or visualisation shouldn’t need to display it. If it is, the end user can then look further into the data or request the report be generated.

Post Processing Searches

You may have a number of panels on your dashboard all running for different base searches…All the searches may be similar and as a result could be duplicating a lot of work across each of the panels. The more panels we add to our dashboard, the more the performance of our dashboard may be impacted and may be slowing down.

This is where post processing searches can help us improve the performance and speed up the display of data on our dashboard, by creating one main or base search for the dashboard. This base search cleans out and transforms our data to minimize the information that is then passed onto our panels, which will use post processing searches to then display the data we want.

Post processing searches are an effective way of conserving resources when you are using a single indexer, but they may not be as effective if you are on an environment that is using multiple indexers. It may be more effective to use multiple searches in the dashboard.

If you would like to learn more about Post Processing Searches and how they can be implemented, Learning Splunk Web Framework is a good resource to get started and help you develop your knowledge.

There are one or two limitations of using post processing in Splunk. The first is that the base search will only return the first 500,000 results, meaning that we may have incomplete data presented to our users. To work around this issue, we need to make sure that our base search is a transformational search, meaning that our base search will strip out a large amount of unwanted data before we move on to present this data further to our user.

Secondly, the post processing search cannot run for longer than 30 seconds and the search will time out and cause an error on your dashboard.

Using Scheduled Reports in Dashboards

Post processing searches are not the only way we can speed up the display of data in our dashboards. We can also use a scheduled report to provide data for the interface we are creating. Using a scheduled report, we also work around some of the limitations that are put in place when using post processing searches, and with large data sets we can run our report once a week and ensure that the data is present before users need to view it.

Found this post useful? Kindly tap the ❤ button below! :)

About The Author

Vince has worked with Splunk for over 5 years, developing apps and reporting applications around Splunk, and now works hard to advocate its success. He has worked as a system engineer in big data companies and development departments, where he has regularly supported, built, and developed with Splunk. He has now published his first book via Packt Publishing — Learning Splunk Web Framework.