Splunk Configuration Files Precedence Explained

When you first look at the way that your configuration files are set up in Splunk it seems like nothing too challenging or difficult. There seems to be a configuration file for everything but there’s nothing that you won’t be able to get your head around.

That’s until you make a change. You restart your server, nothing happens…It could quite possibly be that configuration file precedence has taken effect and your changes are not being picked up as a result.

Hopefully the following points will be able to clear things up a little. Please note that the following is for a standalone installation and does not include a clustered environment.

The Same File In Multiple Locations

When you start working with configuration files in Splunk, you’ll notice that the same configuration file will be located in many different places. Each different location for a configuration file will serve a different purpose and may make changes to users, apps, the entire system and on top of that the default or local system.

Never Change The Default Config File

The default directory for all your configuration files is cleverly in place to show us what a pristine system with no configuration changes would look like. As an administrator, we would never make changes directly to a system without using a configuration management system like Ansible or Puppet, would we? Unfortunately, this does happen, but if you make changes, please don’t touch the default configuration, make a copy of the file and place it in the local directory, and only make changes to this file.

System Directory Is King

When you start to work with your configurations, you will notice that all configurations branch from the etc directory. The main directories that you will be working with are:

  • apps for all application directories and configurations
  • system for configurations that control your entire system
  • users configurations that are specific to each user

When configurations are implemented the precedence that Splunk will use to determine is:

  1. System local directory
  2. App local directory
  3. App default directory
  4. System default directory

But What About Users?

Yep, we’re getting to that. When you through the user configurations into the mix you will then get the following order or precedence:

  1. User directory
  2. App directory, local has priority over default
  3. System directory, local has priority over default.
Remember the users configuration directory does not contain a default directory.

Found this post useful? Kindly tap the ❤ button below! :)

About The Author

Vince has worked with Splunk for over 5 years, developing apps and reporting applications around Splunk, and now works hard to advocate its success. He has worked as a system engineer in big data companies and development departments, where he has regularly supported, built, and developed with Splunk. He has now published his first book via Packt Publishing — Learning Splunk Web Framework.