The Google for Logs [ Splunk ]

Have you heard about a kind of Google for logs?

So, I present you: Splunk!

This is a tool that you want to use alongside your projects and your system, mostly if they have lots of data.

Splunk will be a best friend!

Let’s start!

At this point you can download Splunk from the following link: http://www.splunk.com/en_us/download.html

Access Splunk web interface at port 8000 in your local machine.

Splunk platform works with any data. In particular, it works with all IT streaming and historical data. The source of the data can be event logs, web logs, live application logs, network feeds, system metrics, change monitoring, message queues, archive files, and so on. All of them can be stored by upload via panel:

When you start search by clicking on the green button, Splunk takes you to the page where the magic really begins! \o/

Here you can customize the logs information whatever you want!

• Choose the info that matters.
• Select the period of time that the logs have occurred.
• Change the log type.

And lot of other stuffs.

So let’s start Googling!

The Search bar will be your basis when you become familiar with the syntax language of Splunk. The search assistant could be a great friend to be within the right terms.

You start with the name of the project or something related to a project that you want to search and apply the syntax. For example, to find some error, fail and severe log you can type:

“you project name” (without quotes) (error OR fail* OR severe)

The asterisk (*) character is used to match various terms that begins with the previous word, in this case “fail”. The “OR” is like on any syntax that understands by itself as a simple “or” preposition.

The Events tab displays the Timeline of events, the Fields sidebar, and the Events viewer.

At this one you can select any kind of info you want to be showed by a log.

You can also use Splunk with Docker integration. As the name by itself set, here you’ll use an image provided by outcouldman to add Splunk as a Docker container:

docker pull outcoldman/splunk:latest

or add it manually:

git clone https://github.com/outcoldman/docker-splunk.git
cd docker-splunk/splunk
docker build --tag="$USER/splunk"

You can also start the container manually:

docker run --name vsplunk -v /opt/splunk/etc -v /opt/splunk/var busybox
docker run --hostname splunk --name splunk --volumes-from=vsplunk -p 8000:8000 -d --env SPLUNK_START_ARGS="--accept-license" outcoldman/splunk:latest

You need to run Busybox container for Splunk security container, if it fails, restarts or stops.

Or if you use Docker Compose:

vsplunk:
  image: busybox
  volumes:
    - /opt/splunk/etc
    - /opt/splunk/var
splunk:
  image: outcoldman/splunk:latest
  hostname: splunk
  volumes_from:
    - vsplunk
  ports:
    - 8000:8000

It’s interesting to know about the Splunk ports and your respective functions:

• 8000/tcp — Splunk Web interface (Splunk Enterprise and Splunk Ligh
• 8089/tcp — Splunk Services (All Splunk products)
• 8191/tcp — Application KV Store (Splunk Enterprise)
• 9997/tcp — Splunk Indexing Port (not used by default) (Splunk Enterprise)
• 1514 — Network Input (not used by default) (All Splunk products)
• 8088 — HTTP Event Collector

See more at: https://github.com/outcoldman/docker-splunk

This is the right way to use Splunk as a Docker container. However, we are not in the end yet.

In order to let Splunk reads the docker containers we need to add a Token to HTTP Event Collector via web interface:

Go to: Settings > Data Inputs.

After that click on HTTP Event Collector.

At the top go to Global Settings and select Enabled to All Tokens field. Then click on Save.

Click on New Token, give it a name and follow the steps.

Here we are… Ready to receive logs from docker through this token.

Now we are ready to test the Splunk logging driver. You can configure the logging driver for the whole Docker daemon or per container. For this example, I am going to use the outcoldman/splunk container and configure it for the container:

docker run --publish 80:80\
 --log-driver=splunk\
 --log-opt splunk-token=9B0A57407-BF30–4183-BC9C-0EBC73A6E100\
 --log-opt splunk-url=https://localhost:8000\
 --log-opt splunk-insecureskipverify=true\
 outcoldman/splunk:latest

So let’s send some request to the url to generate some logs output:

curl localhost:80
curl localhost:80?testing-my-splunk-logs

Back to your Search and Reporting Splunk interface and search for the Docker logs by the name you set to your Token:

source = "your-token-name" (with quotes)

Awesome! You can see all the logs with every management possibility that Splunk gives you to analyze and control your data logs.

Hope it can help you to understand a bit about the Google for Logs! SPLUNK! :P