How to send security logs from AWS EC2 Linux hosts to Splunk Cloud

Configuring Splunk Cloud to receive your AWS EC2 Linux security logs is most likely the fastest way to get enterprise-grade logging off the ground.

While Splunk Cloud is a fabulous platform for collecting logs and detecting potential security issues — the documentation on how to make it all fit together is, let’s be diplomatic here, less than perfect.

Had I not already gone bald years ago, trying to configure logging with Splunk Cloud would have been an excellent time pull my hair out and scream in despair -”what have I done to deserve such poor documentation!!!”.

So to spare you a logging config journey to hell and back — here’s how to set up Splunk Cloud logging in less than ten minutes.

In short, we need to do two things:

1: Configure our AWS EC2 Linux server to collect logs locally and send them to Splunk Cloud using a splunkforwarder

2: Configure Splunk Cloud to receive those logs and display them in the Splunk Cloud search interface

Configuring AWS EC2 Linux for log collection

Start by downloading the splunkclouduf.spl config file from your Splunk Cloud account. This file is unique to your Splunk Cloud account and is the key used to authenticate your AWS EC2 Linux splunkforwarder against Splunk Cloud.

Download the splunkclouduf.spl from Splunk Cloud by selecting App: Search & Reporting and then Universal Forwarder like so:

Image for post
Image for post
Getting the universal forwarder from Splunk Cloud

Then click the green Download Universal Forwarder Credentials button:

Image for post
Image for post
Universal Forwarder

With the splunkclouduf.spl locally downloaded, copy it to your AWS EC2 Linux host using scp:

scp -i <yourAWScertificate.pem> ../../Downloads/splunkclouduf.spl :/tmp

Once the splunkclouduf.spl has been uploaded, SSH to your AWS EC2 Linux machine and then download the splunkforwarder

wget -O splunkforwarder-7.1.2-a0c72a66db66-linux-2.6-x86_64.rpm ‘

Install the splunkforwarder

sudo yum localinstall splunkforwarder-7.1.2-a0c72a66db66-linux-2.6-x86_64.rpm -y

Change to root (to make things easier to configure)

sudo su

Export the path to our splunkforwarder installation in /opt/splunkforwarder

export SPLUNK_HOME=/opt/splunkforwarder

Start the splunkforwarder like so:

$SPLUNK_HOME/bin/splunk start

…and assign it a good password

The next step is to configure the splunkforwarder to use our splunkclouduf.spl authentication file we downloaded earlier.

$SPLUNK_HOME/bin/splunk install app /tmp/splunkclouduf.spl

…the user name is admin

We now need to decide what directories our splunkforwarder will monitor and forward to Splunk Cloud. From a security perspective, /var/log/secure and /var/log/audit/audit.log are a must.

$SPLUNK_HOME/bin/splunk add monitor /var/log/secure
$SPLUNK_HOME/bin/splunk add monitor /var/log/audit/audit.log

Now the we save configured what to log, we need to tell splunkforwarder where to send the logs. We do this with the set deploy-poll command:

$SPLUNK_HOME/bin/splunk set deploy-poll input-<your splunk instance>.cloud.splunk.com:8089

Note: it’s easy to mess this step up since you have to prefix your Splunk Cloud instance URI with input-

So if your Splunk Cloud instance is for example prd-p-3jgfwh45rzkz.cloud.splunk.com, the splunk deploy-poll command would then look like so:

$SPLUNK_HOME/bin/splunk set deploy-poll input-prd-p-3jgfwh45rzkz.cloud.splunk.com:8089

Now restart the splunkforwarder using the new configuration:

$SPLUNK_HOME/bin/splunk restart

Everything is now set up on the client side. Now we need to tell Splunk Cloud what to do with the logs it receives.

Configuring Splunk Cloud

Logged in to the Splunk Cloud console, select Settings and Add Data:

Image for post
Image for post
Adding splunkforwarder data to Splunk Cloud

Select forward

Image for post
Image for post

The AWS EC2 Linux host that you just installed the splunkforwarder on should be visible under Available hosts. If it isn’t, wait a few minutes before you click refresh.

Select the AWS EC2 Linux from available hosts and give is a suitable Server Class Name like My server

Image for post
Image for post
Selecting a splunkforwarder server.

Since we’ll be getting /var/log files from our splunkforwarder , select Files & Directories and enter /var/log in the File or Directory field like so:

Image for post
Image for post

Click Next

Leave the Source Type and Index with the default settings:

Image for post
Image for post

Click Review and then Submit

That’s it! Now go the Searching & Reporting view and watch the logs from AWS roll in like colorful rays of light

Image for post
Image for post

Written by

I like abstract art and concrete output

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store