Making sense of container log data

Application logs produce a massive amount of valuable data on how well the application works and also some insight into how users are interacting with it. But making sense of that data might be a bit problematic due to the abundance of data and due to the variation of the data.

One of the great principles in 12factor apps is to:

Treat logs as event streams

And what do you usually do for a stream of data? Yes, you store and index it so you can build some analytic queries on top of the data. That’s exactly what you should do for your logs also.

In my previous blog I explained how Kontena internally gathers and handles the log data of all services running on the Kontena Platform.

EFK Stack

For making sense of your log data, there are a multitude of tools available. In open source one of the most used solution seems to be the EFK stack. Elasticsearch for storing and indexing the data, Fluentd to receive, sanitize and parse the log data and then Kibana to visualize the data for us humans.

In this post, I’m not gonna dive deep into the EFK stack itself, I’m more painting a picture how to tie your apps and the Kontena platform into the EFK stack. And naturally how to setup EFK on Kontena.

The target architecture for our log streaming and analytics setup is:

Why Fluentd and not Logstash as in the “original” ELK stack?

Yes, I actually started with that but ran into some troubles in Logstash parsing out Fluentd protocol messages. Not really sure why though as Fluentd messages are essentially just messagepack messages over the wire. As I didn’t want to really dig down deep into those errors it was natural to switch to Fluentd agent as it has more robust support for Fluentd protocol. D’oh, they’re from the same community. :)

Elasticsearch

Elasticsearch is a wonderful tool to index and manage large sets of semi-structured data. The only downside is that it’s a bit heavyweight as it’s built on top of JVM technologies. So do reserve a bit of memory for the nodes you’re planning to setup Elasticsearch into. The default operating system limits on mmap counts is likely to be too low, which may result in out of memory exceptions. So increase the `max_map_count` on the hosts. This can be done for example with:

kontena node ssh node-1 sudo sysctl -w vm.max_map_count=262144

We’ve made it super easy to get Elasticsearch up-and-running. All you need is kontena stack install kontena/elasticsearch. It will ask you a couple of questions and go and deploy Elasticsearch for you.

By default the stack does not expose Elasticsearch out of the grid, but all other grid apps can connect to it using URL http://elasticsearch.${GRID}.kontena.local:9200.

Elasticsearch by default comes with really easy-to-guess default passwords. The Kontena stack for Elasticsearch is implemented so that during the initial setup it autogenerates new passwords, if asked to do so, and naturally stores them in Vault and updates Elasticsearch to use those. Providing a bit of increased security out-of-box. You’re welcome. :)

Fluentd

Fluentd is the component that receives the log data from the Kontena agent, or other sources, parses and sanitizes it and then stores the information in Elasticsearch for indexing. The Fluentd agent can be installed using kontena stack install kontena/fluentd-elasticsearch. It will ask you a couple of questions and go and deploy Fluentd agents for you.

The Fluentd stack for Elasticsearch is designed so that it actually spins up The Fluentd agent on each node using Kontena’s daemon deployment strategy. This is done for a couple of reasons:

• to balance the processing of logs on all nodes
• to make log processing local on nodes. This saves the overlay network bandwidth a bit.

Kontena Fluentd-Elasticsearch stack is also constructed with built-in handling for the Logstash template for Elasticsearch. What that means in practice is that when Fluentd starts up, it ensures that the needed template(s) are configured in Elasticsearch so that your logs get properly indexed.

Kibana

Kibana is the front-end of the solution. It provides fancy UIs to visualize and query the log information and to create different kinds of dashboards even.

To set up Kibana, again all you need is kontena stack install kontena/kibana. It will ask you a couple of questions and go and deploy Kibana for you and connects it to the Elasticsearch we just set up few moments ago.

Kibana stack is designed so that Kibana will be put behind a loadbalancer of your choosing. That means you can then access Kibana using the given hostname, assuming that DNS points correctly to the Kontena LB nodes.

You can connect to Kibana using username elastic , the password is stored in Kontena vault. You can read the password using kontena vault read --value elastic-password .

Kontena & EFK

Kontena already collects all the service and container logs in Kontena Master database. That allows users to have a single access point for the logs, but it doesn’t provide long-term storage nor any real search or analytics options.

To make the Kontena platform stream all service logs into our new and shiny EFK stack one needs to only enable Fluentd forwarding. That can be done using kontena grid update --log-forwarder fluentd --log-opt fluentd-address=localhost my-grid. Localhost, you ask? Yes, indeed we're streaming all the logs from the Kontena agent to the Fluentd agent on the same node. Thus all nodes are parsing the logs and also we save some precious network bits.

Once you’ve enabled the log forwarding you should see all service and container logs flowing into Elasticsearch and made available in the Kibana front-end. During the first login to Kibana, it asks you to configure an index pattern. The default values are sufficient in this setup and you can just click Create to be able to query the logs.

Summary

Storing container and service logs in queryable form makes lot of sense. It not only provides centralized long-term storage and management for the logs but it also enables humans to make some sense of the log data. We’ve made it super easy to set up your own EFK stack with our ready-made stacks.

About Kontena

Want to learn about real life use cases of Kontena, case studies, best practices, tips & tricks? Need some help with your project? Want to contribute to a project or help other people? Join the Kontena Forum to discuss more about our Kontena Platform, chat with other happy developers on our Slack discussion channel or meet people in person at one of our Meetup groups located all around the world.