Tutorial: collecting and analyzing Docker container logs with Sumo Logic (for free)

Leonid Makarov
Mar 2, 2018 · 6 min read

When it comes to powerful log analytics options for Docker, there are many commercial options out there, but they can be quite expensive. I’m going to share one which offers a free tier without compromising on the features.


Docker has a built-in logging driver called json-file . Container logs are formatted as JSON and written to a text file per container. You use the docker logs command to view the logs as plain text. The only filtering options available are since and until. You may pipe the output to grep to do keyword search, same as with plain text file logs.

This is good enough when you want to view logs for a single container and debug issues ad-hock, right now. For anything more complex (e.g. parsing, filtering and sorting) you‘ll want a real log collection and analytics tool.

There are multiple options out there. Some are paid. Some are free and open source, but DIY. There is one, though, which combines the power and convenience of a commercial SaaS platform with a free entry level plan for small projects and hobbyists — Sumo Logic.

Sumo Logic offers a free plan with 0.5GB/day (15GB/mo) log data ingestion. If you are smart about what goes into your container logs and how often, you’ll be able to stretch the data allowance and get the power of a professional SaaS log collection and analytics platform for free.

Let’s dive in!

Setup

Sign-up for a free trial with Sumo Logic. You will have to use your “work” email address. Consumer mailboxes like Gmail won’t work.

Under Administration > Security > Access Keys click the + icon in the top right corner to add a new key:

Pick the Access Key Label and click Generate Key. Copy and store your keys somewhere secure. You will use these when starting the Sumo Logic Docker Collector container.

Start the collector container on your Docker host

$ docker run -d -v /var/run/docker.sock:/var/run/docker.sock --name=sumo-logic-collector --restart=always sumologic/collector:latest <AccessID> <AccessKey>

Replace <AccessID> and <AccessKey> with the values you recorded previously.

Give the collector a minute to initialize, then go to Manage Data > Collection to confirm it shows up in your account and logs data.

There will be two data sources streamed by the collector container:

  • Docker-stats — CPU/memory/network/etc. container stats
  • Docker-logs — actual container logs

If the collector does not show up in Sumo Logic, then check the collector container logs for clues:

$ docker logs sumologic-collectorRunning SumoLogic Collector...wrapper  | --> Wrapper Started as Console
wrapper | Java Service Wrapper Standard Edition 64-bit 3.5.13
wrapper | Copyright (C) 1999-2011 Tanuki Software, Ltd. All Rights Reserved.
wrapper | http://wrapper.tanukisoftware.com
wrapper | Licensed to Sumo Logic Inc. for Collector
wrapper |
wrapper | Launching a JVM...
jvm 1 | WrapperManager: Initializing...
jvm 1 | . . . . . . . . .
jvm 1 | .+'|=|`+. .+'| |`+. .+'|=|`+.=|`+. .+'|=|`+.
jvm 1 | | | `+.| | | | | | | `+ | `+ | | | | |
jvm 1 | | | . | | | | | | | | | | | | | |
jvm 1 | `+.|=|`+. | | | | | | | | | | | | | |
jvm 1 | . | | | | | | | | | | | | | | | |
jvm 1 | |`+. | | | | | | | | | | | | | | | |
jvm 1 | `+.|=|.+' `+.|=|.+' `+.| |.| |+' `+.|=|.+'
jvm 1 | Sumo Logic Collector Version 19.209-26
jvm 1 | Sumo Logic Build Hash fa2afe3
jvm 1 | current folder:/opt/SumoCollector
jvm 1 | * See /opt/SumoCollector/./logs for more details.
jvm 1 | * Connecting to https://collectors.sumologic.com.
jvm 1 | * ERROR: Registration failed: Your Sumo Logic credentials could not be verified. Make sure the token or accessKey/ID is valid and your user account has permissions to manage Collectors. (error key: collectors.unauthorized)
jvm 1 | Collector exiting...
wrapper | <-- Wrapper Stopped

UTC timezone is used by default for logs timestamps. You may want to adjust that in the settings (Edit link) as necessary.

See the official sumologic-collector-docker repo for additional configuration options and documentation.

Docker stats dashboard in Sumo Logic

Let’s start with some nice dashboards and graphs available out of the box.

Sumo Logic has a “Docker App” available in the App Catalog. Go ahead and add it to your Library.

Set the Source Category for logs to docker

Once the app is installed and some data is collected, you will see some nice graphs pulled for the containers stats.

Searching and analyzing container logs

Now let’s see how we can get to the actual container logs in Sumo Logic.

Create a new Log Search using the “+ New” button in the top right (you can also use the alt+s keyboard shortcut for this)

To only view container logs use the _source=Docker-logs filter. You can also narrow the search down by collector name, source host, etc. Sumo Logic will automatically suggest filter options and available values.

To view logs for a specific container — add the _sourcename filter, e.g.

_source=Docker-logs
AND _sourcename = "plex-server"

To search for a specific log message or keyword, either add it manually in the search query as AND <keyword> filter or highlight it and use the options from the dropdown.

Easy! Now we are only viewing the specific messages we are interested in.

Once we got down to the instances of the messages we are interested in, we may want to see what else was captured in the logs around that time. Sumo Logic makes this extremely easy as well. Click on any of the filters under a message and select Surrounding Messages and a timeframe:

There is so much more that can be done with Sumo Logic. Checkout the docs and video tutorials available on YouTube and in the app.


Disclaimer: I’m not affiliated in any way with Sumo Logic and did not receive any form of compensation from them or anyone else to write this article. I was looking for a solution for my own needs and found Sumo to fit it very nicely.

I hope you enjoyed reading and found this tutorial useful.
Clap all 50 times if so! This helps others discover content on Medium.

HackerNoon.com

how hackers start their afternoons.

Leonid Makarov

Written by

Chief Architect@FFW US, Docksal (docksal.io) creator and maintainer

HackerNoon.com

how hackers start their afternoons.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade