Securing the data in motion from Splunk Universal/Heavy Forwarder to Cribl Host using TLS certs

A series of 4 blogs about Encrypting data in motion between Splunk and Cribl

This is the fourth continuation of posts in the blog series for Encrypting data in motion between Splunk and Cribl. This post deals with encrypting data in motion “from the Splunk UF/HF” as log source/client “to the Cribl host” as destination/server.

If you want to learn more about fundamentals, please click here.

On the high level this is what we are trying to do —

1. Configuring the TLS certs settings in Cribl — Step 1
2. Configuring the TLS certs settings in Splunk Enterprise — Steps 2 & 3 as mentioned below

Step 1—

On Cribl Stream UI, We will go to Sources and look for Splunk TCP which will open the listening port on Cribl alongside defining the TLS settings this is where the cert & key pair are to be defined and added.
The below picture details all the self-explanatory information.

Image 1

This will look like the below once the above configs are saved and the mentioned cert is selected.

Image 2

Step 2—

For configuring the TLS settings on the client Splunk Universal/Heavy forwarder side, we need the certificates to be placed on the system —

We would deal with 2 certificates where —

i. The sslRootCAcert which is the concatenated file with the below sequence —

[ intermediate certificate]
[ certificate authority certificate]

ii. clientCert which is the concatenated file with the below sequence —

-----BEGIN CERTIFICATE-----
... (certificate for your server)...
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
...<Server Private Key – Passphrase protected>
 -----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
... (certificate for ****your**** server)...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
... (the ****intermediate**** certificate)...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
... (the ****certificate authority**** certificate)...
-----END CERTIFICATE-----

Image 3

As part of preparing TLS certificates for use, you must combine them with the private keys that you either received or generated to create the certificates into a single certificate file that the Splunk platform can use.

All certificates and respective keys that you use on the Splunk platform must be concatenated in this manner. A certificate or key alone does not work, even if you configure it correctly. Regardless of the service or contact point you secure, they all must use combined certificates files.

Splunk expects the certs (except CA cert) to be defined in the chain including the serverCert+privateKey+CA cert.

More information link on Splunk docs link below —

How to prepare TLS certificates for use with the Splunk platform

Sometimes the certs are with the .crt extension, those can be changed to the .pem extension using the command mentioned in the screenshot below.
Splunk expects certs/chains in the .pem format.

Image 4

Step 3—

Now, For configuring the TLS settings on the client UF/HF side, we need to update two configuration files —

1. server.conf — this is where we need to define intermediate and root CA certs in the single chain from the step mentioned above.

[sslConfig]
sslRootCAPath = /opt/splunk/etc/auth/mycerts/ca_Cert_Chain.pem
# sslRootCAPath = <Absolute path to the CA certificate. The default value is $SPLUNK_HOME/etc/auth/cacert.pem>

Image 5

Image 6

2. outputs.conf — this is where we need to define the cert chain prepared in step 3A2 as mentioned above along with other attributes as desired.

[tcpout:criblworker]
server=10.1.1.1:9998
disabled = 0
clientCert = /opt/splunk/etc/auth/mycerts/myclient_Cert_Key_CertChain.pem
# useClientSSLCompression = true
# sslPassword = myCertificatePassword
# sslCommonNameToCheck = indexer1.mycompany.com,indexer2.mycompany.com
# sslAltNameToCheck = indexer3.mycompany.com
# sslVerifyServerCert = true

Image 7

Image 8

3. Using the CLI, restart the splunkd process:

# $SPLUNK_HOME/bin/splunk restart splunkd

More information about how we placed such outputs.conf and server.conf configs on the client side can be found in the below link —

Configure Splunk indexing and forwarding to use TLS certificates

Your comments and suggestions are welcomed.