How to send security logs from AWS EC2 Linux hosts to Splunk Cloud

How to send security logs from AWS EC2 Linux hosts to Splunk Cloud
Robert Svensson
Jul 20, 2018·4 min read
Configuring Splunk Cloud to receive your AWS EC2 Linux security logs is most likely the fastest way to get enterprise-grade logging off the ground.
While Splunk Cloud is a fabulous platform for collecting logs and detecting potential security issues — the documentation on how to make it all fit together is, let’s be diplomatic here, less than perfect.
Had I not already gone bald years ago, trying to configure logging with Splunk Cloud would have been an excellent time pull my hair out and scream in despair -”what have I done to deserve such poor documentation!!!”.
So to spare you a logging config journey to hell and back — here’s how to set up Splunk Cloud logging in less than ten minutes.
In short, we need to do two things:
1: Configure our AWS EC2 Linux server to collect logs locally and send them to Splunk Cloud using a splunkforwarder
2: Configure Splunk Cloud to receive those logs and display them in the Splunk Cloud search interface

Configuring AWS EC2 Linux for log collection

Start by downloading the splunkclouduf.spl config file from your Splunk Cloud account. This file is unique to your Splunk Cloud account and is the key used to authenticate your AWS EC2 Linux splunkforwarder against Splunk Cloud.

Download the splunkclouduf.spl from Splunk Cloud by selecting App: Search & Reporting and then Universal Forwarder like so:

Image for post — Getting the universal forwarder from Splunk Cloud

Then click the green Download Universal Forwarder Credentials button:

With the splunkclouduf.spl locally downloaded, copy it to your AWS EC2 Linux host using scp:

scp -i <yourAWScertificate.pem> ../../Downloads/splunkclouduf.spl ec2-user@<your-ec2-host>.compute-1.amazonaws.com:/tmp

Once the splunkclouduf.spl has been uploaded, SSH to your AWS EC2 Linux machine and then download the splunkforwarder

wget -O splunkforwarder-7.1.2-a0c72a66db66-linux-2.6-x86_64.rpm ‘https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.1.2&product=universalforwarder&filename=splunkforwarder-7.1.2-a0c72a66db66-linux-2.6-x86_64.rpm&wget=true'

Install the splunkforwarder

sudo yum localinstall splunkforwarder-7.1.2-a0c72a66db66-linux-2.6-x86_64.rpm -y

Change to root (to make things easier to configure)

sudo su

Export the path to our splunkforwarder installation in /opt/splunkforwarder

export SPLUNK_HOME=/opt/splunkforwarder

Start the splunkforwarder like so:

$SPLUNK_HOME/bin/splunk start

…and assign it a good password

The next step is to configure the splunkforwarder to use our splunkclouduf.spl authentication file we downloaded earlier.

$SPLUNK_HOME/bin/splunk install app /tmp/splunkclouduf.spl

…the user name is admin

We now need to decide what directories our splunkforwarder will monitor and forward to Splunk Cloud. From a security perspective, /var/log/secure and /var/log/audit/audit.log are a must.

$SPLUNK_HOME/bin/splunk add monitor /var/log/secure
$SPLUNK_HOME/bin/splunk add monitor /var/log/audit/audit.log

Now the we save configured what to log, we need to tell splunkforwarder where to send the logs. We do this with the set deploy-poll command:

$SPLUNK_HOME/bin/splunk set deploy-poll input-<your splunk instance>.cloud.splunk.com:8089

Note: it’s easy to mess this step up since you have to prefix your Splunk Cloud instance URI with input-

So if your Splunk Cloud instance is for example prd-p-3jgfwh45rzkz.cloud.splunk.com, the splunk deploy-poll command would then look like so:

$SPLUNK_HOME/bin/splunk set deploy-poll input-prd-p-3jgfwh45rzkz.cloud.splunk.com:8089

Now restart the splunkforwarder using the new configuration:

$SPLUNK_HOME/bin/splunk restart

Everything is now set up on the client side. Now we need to tell Splunk Cloud what to do with the logs it receives.

Configuring Splunk Cloud

Logged in to the Splunk Cloud console, select Settings and Add Data:

Select forward

The AWS EC2 Linux host that you just installed the splunkforwarder on should be visible under Available hosts. If it isn’t, wait a few minutes before you click refresh.

Select the AWS EC2 Linux from available hosts and give is a suitable Server Class Name like My server

Since we’ll be getting /var/log files from our splunkforwarder , select Files & Directories and enter /var/log in the File or Directory field like so: