Feeding Heroku logs to Splunk

Having recently developed an application on the PaaS Heroku I decided to experiment a bit by setting up my own instance of Splunk Enterprise (Free edition) and feeding my application’s logs to Splunk Enterprise.

Note: this isn’t meant to be a professional guide of any kind, but rather a brief description of my experience, in the hope that it may help someone, someday.

Step 1 (Decide where to host your Splunk)

I personally created a Digital Ocean droplet with very little resources, given the educational nature of my experiment. I am talking about 1 GB / 25 GB Disk. The host runs Ubuntu 20.04 (LTS) x64.

You might want to opt for more resources, depending on your needs and the amount of data you expect it to ingest.

Step 2 (Install the Free edition of Splunk)

To do so, I opened Splunk’s website, clicked on the“Free Splunk” link and filled out the form to establish an account. I was then offered a choice of download methods and I chose the Download via CLI.

wget command for CLI download

Step 3 (Install Splunk and set it up)

I then opened a terminal in my droplet and ran the following commands

Download Splunk

wget -O splunk-8.2.2–87344edfcdb4-linux-2.6-amd64.deb ‘https://d7wz6hmoaavd0.cloudfront.net/products/splunk/releases/8.2.2/linux/splunk-8.2.2-87344edfcdb4-linux-2.6-amd64.deb'

Move the deb file to tmp

mv splunk-8.2.2–87344edfcdb4-linux-2.6-amd64.deb /tmp

Open tmp

cd /tmp

Install (it took a few minutes for me, given the little resources)

sudo dpkg -i splunk-8.2.2–87344edfcdb4-linux-2.6-amd64.deb

I then chose to start Splunk at boot

sudo /opt/splunk/bin/splunk enable boot-start

After accepting the license terms, I set up an administrative account

License Terms and Admin setup

I then started the Splunk service

sudo service splunk start

I opened the GUI, to verify the service was running and functional

Splunk Dash — Connection not secure

By default, Splunk runs on HTTP, but SSL can be easily enabled.

Step 4 (Enable SSL)

To enable SSL on the Splunk GUI, I selected Settings > System > Server settings, and then clicked General Settings.
Under Splunk Web, I selected the Yes radio button.

Enable SSL

I restarted the Splunk service, and noted that SSL was now active.

Splunk running on HTTPS

Step 5 (Enable Syslog Data Input)

I then enabled Data input for my instance of Splunk, which acts as an Indexer and Search Head in my very basic architecture. From the GUI, I selected Settings > Data Input > TCP and configured as below to enable receiving data via syslog

Data Input configuration — part 1

Data Input configuration — part 2

I verified that the service was listening via console

netstat -pnltu | grep 514

Listening on Syslog port

Step 6 (Configure Heroku Logging)

I issued the below commands via the Heroku CLI

Enable Debug Logs

heroku config:add LOG_LEVEL=DEBUG --app <APPNAME>

Runtime metrics

heroku labs:enable log-runtime-metrics --app <APPNAME>

I restarted the app

heroku restart --app <APPNAME>

Step 7 (Configure a log drain)

I then configured a syslog drain to send my application’s logs to my indexer

heroku drains:add syslog://<INDEXER’S IP>:514 --app <APPNAME>

Confirmation of Drain creation

Step 8 (Verify log data)

After having completed all the steps, logs started flowing to my Indexer and I could start enjoying the power of Splunk.

Logs available in Splunk to search