How to send security logs from AWS EC2 Linux hosts to Splunk Cloud
Configuring Splunk Cloud to receive your AWS EC2 Linux security logs is most likely the fastest way to get enterprise-grade logging off the ground.
While Splunk Cloud is a fabulous platform for collecting logs and detecting potential security issues — the documentation on how to make it all fit together is, let’s be diplomatic here, less than perfect.
Had I not already gone bald years ago, trying to configure logging with Splunk Cloud would have been an excellent time pull my hair out and scream in despair -”what have I done to deserve such poor documentation!!!”.
So to spare you a logging config journey to hell and back — here’s how to set up Splunk Cloud logging in less than ten minutes.
In short, we need to do two things:
1: Configure our AWS EC2 Linux server to collect logs locally and send them to Splunk Cloud using a splunkforwarder
2: Configure Splunk Cloud to receive those logs and display them in the Splunk Cloud search interface
Configuring AWS EC2 Linux for log collection
Start by downloading the splunkclouduf.spl config file from your Splunk Cloud account. This file is unique to your Splunk Cloud account and is the key used to authenticate your AWS EC2 Linux splunkforwarder against Splunk Cloud.
Download the splunkclouduf.spl from Splunk Cloud by selecting App: Search & Reporting and then Universal Forwarder like so:
Then click the green Download Universal Forwarder Credentials button:
With the splunkclouduf.spl locally downloaded, copy it to your AWS EC2 Linux host using scp:
scp -i <yourAWScertificate.pem> ../../Downloads/splunkclouduf.spl ec2-user@<your-ec2-host>.compute-1.amazonaws.com:/tmp
Once the splunkclouduf.spl has been uploaded, SSH to your AWS EC2 Linux machine and then download the splunkforwarder
wget -O splunkforwarder-7.1.2-a0c72a66db66-linux-2.6-x86_64.rpm ‘https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.1.2&product=universalforwarder&filename=splunkforwarder-7.1.2-a0c72a66db66-linux-2.6-x86_64.rpm&wget=true'
Install the splunkforwarder
sudo yum localinstall splunkforwarder-7.1.2-a0c72a66db66-linux-2.6-x86_64.rpm -y
Change to root (to make things easier to configure)
Export the path to our splunkforwarder installation in /opt/splunkforwarder
Start the splunkforwarder like so:
…and assign it a good password
The next step is to configure the splunkforwarder to use our splunkclouduf.spl authentication file we downloaded earlier.
$SPLUNK_HOME/bin/splunk install app /tmp/splunkclouduf.spl
…the user name is admin
We now need to decide what directories our splunkforwarder will monitor and forward to Splunk Cloud. From a security perspective, /var/log/secure and /var/log/audit/audit.log are a must.
$SPLUNK_HOME/bin/splunk add monitor /var/log/secure
$SPLUNK_HOME/bin/splunk add monitor /var/log/audit/audit.log
Now the we save configured what to log, we need to tell splunkforwarder where to send the logs. We do this with the set deploy-poll command:
$SPLUNK_HOME/bin/splunk set deploy-poll input-<your splunk instance>.cloud.splunk.com:8089
Note: it’s easy to mess this step up since you have to prefix your Splunk Cloud instance URI with input-
So if your Splunk Cloud instance is for example prd-p-3jgfwh45rzkz.cloud.splunk.com, the splunk deploy-poll command would then look like so:
$SPLUNK_HOME/bin/splunk set deploy-poll input-prd-p-3jgfwh45rzkz.cloud.splunk.com:8089
Now restart the splunkforwarder using the new configuration:
Everything is now set up on the client side. Now we need to tell Splunk Cloud what to do with the logs it receives.
Configuring Splunk Cloud
Logged in to the Splunk Cloud console, select Settings and Add Data:
The AWS EC2 Linux host that you just installed the splunkforwarder on should be visible under Available hosts. If it isn’t, wait a few minutes before you click refresh.
Select the AWS EC2 Linux from available hosts and give is a suitable Server Class Name like My server
Since we’ll be getting /var/log files from our splunkforwarder , select Files & Directories and enter /var/log in the File or Directory field like so:
Leave the Source Type and Index with the default settings:
Click Review and then Submit
That’s it! Now go the Searching & Reporting view and watch the logs from AWS roll in like colorful rays of light