ELK + Filebeat for Container Logs

“ELK” is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server‑side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a “stash” like Elasticsearch. Kibana lets users visualize data with charts and graphs in Elasticsearch.

Filebeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on your servers, Filebeat monitors the log files or locations that you specify, collects log events, and forwards them either to ElasticSearch or Logstash for indexing.

The picture shows Filebeat sending container logs to logstash, which further sends it to Elasticsearch and we can view it in Kibana.

All the above definitions/statements were from their official website. Now let us come to the important part of configuration.
A big thanks to https://github.com/deviantony/docker-elk , for the boiler plate on how to setup the ELK stack.

The additions would be configuring Filebeat with the ELK stack and make the configuration to take logs from your containers via filebeat, pass it to the logstash , which inturn passes it to ElasticSearch.
You can find the whole configuration here — https://github.com/Shrinidhikulkarni7/docker-elk

Best part of this is everything(ELK + Filebeat) will be configured under a single yml file and will be setup just by a single command (docker-compose up).

Filebeat Configuration

You need to configure the filebeat so that it takes input from containers.
You can do this by using a simple yml file,

In the above snippet you take logs from all the containers by using ‘*’. You can also change it depending on your use case.
The output is sent to logstash which is hosted on port 5000.

Logstash Configuration

When the filebeat sends logs input to logstash, the logstash should be configured to take input from filebeat and output it sent to elastic search.

This can be configured in logstash.conf as following,

In the above snippet, the input to logstash is sent to the beats(filebeat) at port 5000 and the output is sent to elasticsearch which is being hosted at 9200.

This is a simple configuration which makes your life easier to deal with logs and maintaining them.
One other configuration would be to increase the amount of memory used by the services by JVM tuning.
I recommend 1GB to both Elastic and Logstash, but it will all depend on your configuration.

Thank you!