From Splunk 5.x to 7.0: Just what have you been missing out on?
Last year in our Splunk 5.x EOL post we talked about why you should upgrade your Splunk version and explained the main support and security reasons for looking at doing an upgrade.
In this post, we’ll cover the notable new features that have been added since 6.0, as well as the features and platforms that have been removed along the way.
It’s worth noting that Splunk 6.0 was released on 1 October 2013 — over four years ago! And Splunk has changed a lot in those four years. If you’re still sitting on your version 5.x, have you spared a minute to think about all the amazing features you might be missing out on by not upgrading?
We’ve tried to keep our list short (which is difficult with all of the considerable improvements since 2013) but here are the main noteworthy features from Splunk 6.x all the way to 7.0 that you may wonder how you ever did without.
We’ve also included the versions these features were added so that you can see how each upgrade would benefit you.
1. A new UI (6.0)
From 5.x to 7.0, the Splunk User Interface (UI) changed dramatically, moving from a HTML table-driven UI to one based on modern development techniques and frameworks. This resulted in a much better user experience and browser performance and built the capabilities for some of the new visualisations detailed below in point three.
Dashboard editor (6.1)
The new simple XML and UI-driven dashboard creation allowed users to easily create dashboards without having to get their hands dirty with XML. This opens up dashboard creation to the majority of Splunk users, regardless of skill level.
New homepage (6.2)
As with the redesigned dashboard above, the new homepage in 6.2 provided a much cleaner look and listed all of the installed applications available to the user.
Searches can now be done based on technology or app groupings and there is easier navigation between dashboards. A specific dashboard can also now be displayed to all users, be it the licensing overview, environment health or a sales figures dashboard your teams have created.
2. Clustering all the things!
The biggest change in the 6.x cycle for your infrastructure and deployment topology was clustering. Bringing in a completely new model of how your data could be distributed and shared across your infrastructure and, also, how to start horizontally scaling your Splunk instance to improve performance.
Clustering is a major architectural change for Splunk and now a common topology model commonplace in many customer deployments. It grants the capability to scale from 100Gig deployments up to multi-terabytes of scale. This is achieved with a straightforward horizontal scaling model in line with data ingest and user search demands.
Multisite indexer clustering (6.1)
Multisite indexer clustering created a DR-type (disaster recovery) model for data replication within your Splunk environment; It massively reduced the dependency of storage snapshots or backups to ensure that data was recoverable in a failure scenario.
Multisite, essentially, allows for the replication of entire datasets onto additional indexer instances. This can also improve search performance by creating multiple copies of searchable data horizontally across the environment, allowing the search heads to query multiple indexer instances in parallel for the same data.
Search head clustering (6.2)
Search head clustering gave us the same benefits of horizontal scale and resilience at the search head layer. Configured slightly differently to the indexer cluster model with a cluster master, it uses the RAFT protocol to select a captaincy process to manage the synchronisation of data and search results across the cluster.
A great explanation of RAFT and its benefits can be found here: http://thesecretlivesofdata.com/raft/
3. Data visualisation improvements
Splunk’s data visualisation capabilities have increased massively over the 6.x cycle, turning it into a perfect tool to use for real data context, 10ft monitoring displays, management reports and visual investigations into your data.
MAPS! (6.0)
Version 6.0 brought the ability to add geographical context to searches and dashboards by embedding maps natively in Splunk. Native maps allow you to specify custom tile backgrounds and geo-locate information such as IP addresses by longitude and latitude. This allows for powerful visualisation of location-based tracking and the reporting of systems access or information density by country and/or state using a choropleth map.
Dynamic drilldowns (6.1)
While not obvious at first look, dynamic drilldowns can massively change the way users adopt and use dashboards for active investigation of incidents and data.
Dynamic drilldowns allow panels in dashboards to become interactive. You can select a slice of the data you are interested in to power an entirely separate dashboard, or reload the existing dashboard with the changed context.
For example, clicking on the above charts could take you through to another dashboard displaying a deepdive view of the data or reveal additional panels in the current dashboard with additional context to further analyse and improve the users understanding of a fault or a report.
Custom Visualisations (6.4)
As well as the major change from flash-based visualisations to HTML canvas, a new integration model was created allowing for additional visualisations to be added via apps. A number of ‘official’ custom visualisations have been developed, as well as third party ones.
These are the ones you’ll wonder how you lived without:
- Horizon charts: https://splunkbase.splunk.com/app/3117/
Horizon charts give you an awesome positive and minus visual analysis of data with two distinct groupings. You can use it to visualise behaviour, such as performance metrics, across numerous hosts, stock changes, sales orders and anything else you can think of.
- Sankey diagrams: https://splunkbase.splunk.com/app/3112/
Sankey diagrams allow for identification of relationships and values of your dataflows, such as website traffic flows or relationships between systems and API calls.
- Status indicators: https://splunkbase.splunk.com/app/3119/
Status indicators give a visual way of enhancing a single value field in Splunk. This allows for a colour, value and icon to demonstrate the statistics you wish to show in a meaningful way, adding additional context.
- Timeline: https://splunkbase.splunk.com/app/3120/
Timelines allow for a duration and time-based visualisation of your data. It allows you to identify the duration of tasks and activities within your data and visualise it to show behaviours and trends, spot log running processes, downtime or patterns of jobs running on your infrastructure.
The Treemap visualisation displays data in rendered block sizes correlated to the percentage value of the results — a visual way of understanding the percentage breakdown of the data. It is great for understanding market segments, device usage patterns for websites, sales types and more.
- Horseshoe indicator: https://splunkbase.splunk.com/app/3166/
Horseshoe indicators allow KPIs to be visualised against a set of ranges or a target value. They’re ideal for scenarios like reporting on sales or tracking CPU usage and uptime percentage.
- Location tracker: https://splunkbase.splunk.com/app/3164/
Building on from the addition of Maps in 6.0, location tracker allows for location-based updates in your data to be visualised across a map, giving the ability to track routes of vehicles or deliveries, or other location-sensitive data on maps.
There are also a number of other custom visualisations officially supported by Splunk that are suited for different business cases:
- Calendar heatmap: https://splunkbase.splunk.com/app/3162/
- Parallel coordinates: https://splunkbase.splunk.com/app/3137/
- Bullet graph: https://splunkbase.splunk.com/app/3144/
- Punchcard: https://splunkbase.splunk.com/app/3129/
Using these different visualisation functions is as simple as downloading and installing the Splunk app onto your search heads and consulting the built-in app overview page for dashboard examples and use cases.
The framework supporting these allows for any JavaScript-based visualisation to be delivered in Splunk; even developing your own.
Trellis Views (6.6)
In 6.6, trellis views dramatically changed the way we built dashboards. Rather than having to create almost identical panels with additional searches just to look at the difference between a day, an error message or a sales category, we were given the option to chart that data in just one search and, instead, configure trellis mode for the type of visualisation we wish to see.
This allows us to split the visualisation based on the field and automatically creates multiple panels showing the data breakdown. This can be very useful for doing month-to-month views without needing to hardcode the month or KPIs based on web services and other factors.
Chart enhancements (7.0)
The enhancements to charts in 7.0 brought additional tweaking to the visual presentation of your data. A number of new charting options now exist to manage things like the legend, line thickness, dash styles and more.
These are the new options added to charts in 7.0
Chart annotations (7.0)
Chart annotations allow for additional context or information to be added to charts that are on displays. These are added by utilising a secondary search against the chart panel. Further details about how this works under the hood can be found here:
4. Forwarder management UI (6.1)
The deployment server capabilities (otherwise known as the forwarder management UI) added in 6.1 allow for centralised app and configuration management of your forwarder estate.
The universal forwarders call home to the deployment server and will download apps and configuration automatically, keeping themselves in sync with any changes. This means that managing a large forwarder estate becomes a much simpler and straightforward process.
You can group your forwarders into server classes which then inherit applications to be deployed to matching instances. Matches are made from a hardcoded list or through regex style matching of hostname, IP address, etc.
This solution can scale from one or two forwarders into 1000s of forwarders, all being centrally managed from within your core Splunk deployment. Teaming this with an orchestration solution such as Puppet, Chef or Ansible to manage the deployment of the universal forwarder package can transform how Spunk is managed in your business.
5. Bye bye S.O.S; Run DMC! (6.2)
Supporting a Splunk environment (especially a distributed deployment) can be a difficult job without the right tools. During the 5.x releases, we had Splunk on Splunk, or S.O.S for short. While a useful tool for configuration insights and debugging performance or data on-boarding issues, it had one fatal flaw: It consumed part of your Splunk license to use. The way in which S.O.S gathered information meant it ate away at your data ingest, data storage and more.
While useful, it was limited in what it was able to do.
With the 6.2 release came the Distributed Monitoring Console (DMC); now called the Monitoring Console from 6.5 onwards. This solution is designed to run on a dedicated instance for monitoring your Splunk environment’s health and performance. The first changes are visible in the UI elements. It is a much cleaner and straightforward application that can monitor the health of all instances in the Splunk environment. Most critically, it doesn’t consume any of your Splunk license to run as all of its searches are based on the REST API interface and SPL REST command.
This means it is able to gather much more data and information about performance than S.O.S was ever able to compile while being less taxing on your Splunk environment.
6. Data models & pivot (6.x)
Report accelerations had been an important first step in summarising raw data in a reusable and granular way that was easier and more repeatable than summary indexing.
Data models introduced a way to address the summarised data as objects, allowing someone to interact with data without having to learn the search language. It also enabled the pivot interface to be used for rapid creation of common visualisations of data, with the complex search language being hidden from the user.
By using the principles of the report acceleration idea against these new data models, and using the high performance analytics store, significantly faster search performance was suddenly possible. The data-model accelerations (like report accelerations) ran on the search heads and kept their summaries local.
*NB- Data Model Acceleration = Massive Search Performance improvement
Pivot tables introduced a new way of creating dashboards and panels as well as exploring data in Splunk.
Rather than using SPL to search the data, pivot provides an interface more familiar to a business intelligence (BI) or marketing intelligence (MI) tool to slice the data model fields into different views and visualisations. These reports can also be included and placed onto dashboards, creating hybrid views which could contain some panels driven by SPL and some by pivot. This capability opens Splunk up to new users who are already familiar with other BI/MI tools on the market to start gaining new insight into existing data sets.
7. Splunk Analytics for Hadoop (6.5)
Splunk Analytics for Hadoop (originally known as Hunk) allows Splunk to be used as a search UI for data held in Hadoop systems. It offers the same search experience as Splunk Core and a different licensing model based on the scale of your Hadoop deployment. Recent updates have added capabilities around data model acceleration and further search performance improvements around the HAR data format.
This capability allows customers who already have a large data lake solution using Hadoop to still gain benefit and features from Splunk to query, correlate and investigate that data source.
8. Machine Learning Toolkit (6.4–7.0)
With the release of Splunk 6.4 also came the Machine Learning Toolkit (MLTK) 1.0 release (https://splunkbase.splunk.com/app/2890/) allowing you to start applying supervised machine learning (ML) models and training to your Splunk data.
These models could be used to create predictions on your datasets and correlations of data over large data samples for classification type problems.
With the 7.0 release, ML is now more integrated across Splunk than ever before.
The tighter integration gives you the capability to manage your datasets within Splunk and link into systems like Apache Spark to ensure stellar performance for building and training your ML models. It also provides a custom API to allow you to build your own ML algorithms beyond those provided in the app and allow the MLTK to be customised to your bespoke needs.
Additionally, the MLTK will further integrate into IT Service Intelligence 3.0 (which we’ll discuss below), allowing for KPIs to be built based on ML models and trends.
9. Metrics (7.0): The end of ‘meh’-trics
Metrics is a completely new type of data source, based on CollectD and StatsD protocols, that you can now start to ingest into Splunk. It takes in metric data about Internet of Things (IoT) devices, containers, infrastructure and more into a specialised and optimised index, allowing you to run searches about key KPIs in your data. Splunk has seen increases in search performance between 150 per cent and 200 per cent when querying data via metrics versus standard SPL event log queries.
This new feature, teamed with a teased new Splunk APM tool coming out before the end of 2018, will revolutionise and allow parallel investigation of machine and metric data due to be released soon under the following project names:
- Splunk Project Waitomo: A new infrastructure monitoring solution that unifies logs and metrics, delivering integrated machine learning for alerts, trends and investigation.
- Splunk Project Nova: An API-based logging-as-a-service solution, targeting developers and DevOps practitioners.
The image from Splunk .conf showed the same data being queried using metrics versus event data. Looking at more than four million events, the metric search completed in less than a third of a second whereas the event search took over 40 seconds.
10. Premium apps (6.x — 7.0)
Finally, we arrive at number ten on our list. As part of the 6.x series of releases we’ve seen Splunk move beyond being just another machine data analytics platform. Thanks to advancements in areas like ML, Hadoop, data models and pivot, it has grown to become a true data hub and BI/MI tool. But that’s not all.
On top of Splunk we now have newly built solutions. This is not just to get data in but to solve real business problems and address challenges, both in the IT operations and security spaces.
The following apps have allowed Splunk to pivot into other domains and provide major benefit to those customers who adopt these additional capabilities.
IT Service Intelligence (ITSI)
ITSI provides a KPI-driven view into your Splunk data. Version 3.0 has just been announced, adding new capabilities around alerting response action automation, glass table improvements and built-in response playbooks for how to manage alerts and react accordingly; all right from within Splunk.
Enterprise Security
Enterprise Security is a SIEM solution built to run on top of Splunk. It takes advantage of the performance and scaling improvements we’ve mentioned as well as the data models and pivots under the hood. Sitting at the top of the Gartner Magic Quadrant for SIEM technologies, it has beat out IBM, LogRhythm, ArcSight and others to be seen as the leading and most visionary SIEM on the market:
It can transform your Splunk environment and existing data sets into a completely new solution by providing security insight and investigation capabilities, as well as the capabilities to leverage as a fully matured 24/7/365 SOC solution.
Honourable mentions
We could have spent much longer listing off the new features you will benefit from by upgrading your Splunk version, but I hope that our top ten have helped convinced you to move on from your 5.x or 6.x version. If not, here are some other noteworthy features we didn’t have time to go into:
- New search commands and queries: tstats, geostats, iplocation, sendemail, cluster, anomalydetection
- KVStore
- Search optimisations
- HTTP event collection
- SSO SAML 2.0
- Custom alert actions
- Data integrity controls
- Easy Splunk app installs
- Datasets
- New search assistance and syntax highlighting
- Dashboard editor previews
- Forwarder load balancing
- Adaptive response framework
Headline changes version-by-version
From the detail above, you can see that there have been a large number of changes since version 5.0 was released. Here, we will briefly explore the headline new features for each major and minor version, as well as the features removed. This should give you a good overview so that you can assess for yourself the opportunities that an update would offer you.
Splunk 5.0: Released October 2012 (EOL November 2017)
5.0 was a significant release compared to the 4.x releases. It introduced several key new features, deprecated some long-standing favourites and positioned Splunk for future growth and adoption at scale.
Notable new features
Splunk 6.0: Released October 2013
Splunk version 6.0, released a year after version 5.0, took many of the features that v5 had introduced and significantly improved them. It also saw the removal of features and ending of support for some platforms that earlier versions had supported or contained. This formed the firm foundation that Splunk has iterated upon between 2013 and 2017, up to the introduction of Splunk 7.
Notable new features
Notable removed features & platforms
Support for various older platforms was removed for Splunk Enterprise from this version onwards. Earlier versions of the forwarders are still available to gather data and forward, but support for the following OS and CPU platforms was ended: WinXp & Vista x86 & x86_64; Solaris 8 & 9 x86, x86_64, SPARC; FreeBSD 6 x86, x86_64; AIX 5.2, 5.3; Linux kernel 2.4 x86; Mac OS X 10.5, 10.6; HP-UX 11i v2, v3 for PA-RISC
Flash charts were removed as a feature from SimpleXML from 6.0 onwards and several deprecated AdvancedXML models were removed, alongside some other features that were removed because they’d been superseded by newer features.
Splunk 6.1: Released May 2014
Splunk version 6.1 was released six months after 6.0, bundling many improvements on top of the firm foundation that 6.0 had provided.
Notable new features
Notable removed features & platforms
No platforms were removed. Support for Internet Explorer 7 and 8 as browsers was removed. The only feature removed was the multi-tenant deployment server.
Splunk 6.2: Released October 2014
Six months after 6.1, Splunk released 6.2, again mostly iterating on a successful 6.x branch and introducing some key new features.
Notable new features
Notable removed features & platforms
Support for various older platforms was removed for Splunk Enterprise from this version onwards. Earlier versions of the forwarders are still available to gather data and forward, but support for the following OS and CPU platforms was ended: Win7 & 8 x86, x86_64; Win Server 2003 x86, x86_64; Solaris 10 x86–32; FreeBSD 7 & 8 x86, FreeBSD 7 x86_64; Mac OS X 10.7; HP-UX 11i v2, v3 for Itanium
IT data block signing, audit signing and event hashing were all removed. Later versions introduced some alternatives, but if these are essential for your current design, you may not want to move beyond this version.
PDF report server (the legacy version) was finally removed.
Splunk 6.3: Released September 2015
Splunk version 6.3 was released five months after 6.2 and most of the changes were to optimise handling of data behind the scenes.
Notable new features
Notable removed features & platforms
Support for various older platforms was removed for Splunk Enterprise from this version onwards. Earlier versions of the forwarders are still available to gather data and forward, but support for the following OS and CPU platforms was ended: Win 7 & 8 x86_64; Win Server 2003 x86, x86_64; Solaris 10 x86–32; FreeBSD 8 & 9 x86_64; Mac OS X 10.8; AIX 6.1 & 7.1; Solaris 10 x86–32, 10 & 11 SPARC.
Reiser3 was removed as a supported file system.
Splunk 6.4: Released April 2016
Released in spring 2016, Splunk 6.4 introduced some key new features that have helped set the scene for releases throughout 2016 and 2017.
Notable removed features & platforms
Support for various older platforms was removed for Splunk Enterprise from this version onwards. Earlier versions of the forwarders are still available to gather data and forward, but support for the following OS and CPU platforms was ended: Win Server 2008 x86, x86_64; FreeBSD 8 x86; Linux kernel 2.6 x86 (32b), HP-UX 11i v2 Itanium-64.
Internet Explorer 9 and 10 were removed as supported browsers.
AdvancedXML viewer support through the ?showsource=advanced request was removed, alongside the googlemapsview and d3chartview. SimpleXML advancements had rendered AdvancedXML redundant by 6.4, while the two views had been surpassed by new built-in options and the new modular visualisation framework.
Splunk 6.5: Released September 2016
A lot of enhancements to existing features were introduced in 6.5.
Notable removed features & platforms
Support for Mac OS X 10.9 was removed for Splunk Enterprise from this version onwards.
<list> element support was removed from dashboards, having been deprecated since 6.2.
Splunk 6.6: Released March 2017
By 6.6, the 6.x branch was maturing. While 6.6 didn’t introduce masses of new tentpole features, it did introduce some key and valuable enhancements over 6.5.
A new layout method called trellis in dashboards made creating sub-panels from a common search easy, and two new search commands (union and in) were added. Union joins two searches more efficiently than append or join, and in provides an SQL-like search function to be able to easily search for a value in a list of others.
The one major feature that came in 6.6 is the massively improved search optimiser, which parses a search much more efficiently when submitted. This analyses the order of search clauses and rearranges them to run the search most optimally without changing the result.
Notable removed features & platforms
Support for AIX 6.1 was removed for Splunk Enterprise from this version onwards.
How we can help
NCC Group is a long-standing Splunk Partner in UK and EMEA. Our team of highly capable, experienced and qualified Splunk consultants can help you to get the most intelligence and information out of your data.
The team has delivered stand-alone and integrated transformational security, operational and business intelligence services to hundreds of our own and partnered Splunk customers.
We work with everyone — from global enterprises to small start-ups — at data volumes of less than one gigabyte per day to multiple terabytes per day.
If your organisation is already using Splunk, a good starting point is our Health Check service. A Splunk Health Check from NCC Group will tell you the state and maturity of your current deployment, and we’ll highlight what improvements could be made and how things could run more efficiently. On top of that, we’ll detail the value you’d see from upgrading, as well as how you may benefit from putting more data in or handling the data you already have in Splunk differently.
If your organisation isn’t already using Splunk, then talk to us about a demonstration and a discussion of the possible business value it could bring to you. Please email us at response@nccgroup.trust for further details.
