Grafana Loki in Docker Swarm

After I had a couple of hard days with Grafana Loki because there isn’t any available official documentation and the blog posts and tutorials that are created about how can set up and use in Docker Swarm are outdated and not using the latest version of Loki and Grafana. I decided to create this complete step-by-step solution based on my experience how to use Grafana Loki as a log collector in a Docker Swarm cluster. I’m not that person who can write a lot about nothing, so let’s take a deep dive into it right away.

What prerequisites do you need before setup Loki based logging? You simply need to set up a pure Docker Swarm cluster (If you are not sure how to do it, Docker has an easily understandable guide for it: https://docs.docker.com/engine/swarm/swarm-tutorial/create-swarm/)

I think the easiest way is to use the Docker plugin rather than promtail, so foremost we need to install it.

$ docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

In the next step, we need to create a config file for Loki. In a Docker Swarm environment, I prefer to use the config managed by the Swarm rather than a local file on the server, so I’ll use this in this case as well. If for some reason you disagree with me or don’t like to use the Swarm-managed config files, then feel free to use a local config file instead, but in that case, watch out, you’ll have to modify the compose files as well. Create a config file with the content below:

$ wget https://gist.githubusercontent.com/schneidermr/8906c369e9a974154b06d983ab1da9bf/raw/2db88eec44dfb6a2d4736a11ea36c2742a903220/loki-config.yaml
$ docker config create loki_config loki-config.yaml

This is a minimum set of the config for Loki, feel free to extend it and personalize more for your need.

Here is the documentation of Loki configuration: https://grafana.com/docs/loki/latest/configuration/#common_config

The only thing that you need to pay attention to, keep lines #19 and #20 unchanged. Unfortunately, the documentation of Loki is a bit unclear, this frontend.address option is missing, but it’s important to change it because using the default will cause a grpc error and Grafana be never informed of the result of the query — so more understandable without this line of config Grafana never could query the logs in Loki, so the integration won’t work.

Now the only thing we have to do before starting using Loki is to start a container for it. Use the compose below to start the necessary Swarm stack.

$ wget https://gist.githubusercontent.com/schneidermr/b4ff210d163553d76306ba0ead6de7cb/raw/3906fc74cb5fe4b43c07f8c102b246c92c87b8b6/grafana-loki-stack.yml
$ docker stack deploy -c grafana-loki-stack.yml loki

Loki needs to be exposed to a host port to work properly, so for security reasons, it might be important to protect that port from the outside. (If you plan to use Loki to collect logs also from external systems, then it is recommended to setup the authentication layer to protect Loki from unauthorized access and flooding attacks.) First, we need to get the name of the default network interface

$ ifconfig -s -a

In my case it is eth0 so I’ll use it in the following iptables command.

$ iptables -I DOCKER-USER -i eth0 -p tcp --destination-port 3100 -j REJECT

(If you are not experienced in how to use iptables rules let me help and give you some cheat sheet. We will insert a new rule for a DOCKER-USER chain that is -I DOCKER-USER means, for the eth0 network adapter -i eth0 and only for TCP protocol -p tcp. The protected port will be what Loki is using — destination-port 3100 and if a pocket matches with the previous query, the firewall needs to drop that pocket -j REJECT.)

Hurray, we are nearly done, we have both a running Grafana instance that is listening on port 3000 and a Loki instance listening on port 3100. Last but not the least, login to Grafana and configure the Loki data source. As Grafana and Loki share with the same network, we can use the service name in the data source.

In the last step, let’s test if everything working fine with a sample service. Start a basic Nginx instance and configure that to send all its logs to Loki using the installed Docker plugin.

$ wget https://gist.githubusercontent.com/schneidermr/086fac831bbf9f46cbb789b5df7dd025/raw/52c66b8afb373cc75c48c6a97b42419f0a3876a0/sample-service-stack.yml
$ docker stack deploy -c sample-service-stack.yml test

We have a running basic Nginx instance that is listening on port 8080, let’s take a look at it. Open up in a browser window and after that check, Grafana queries the logs from Loki.

Loki uses some labels by default to tag the logs, so we can use it for the query. In my case, I filter for the name of the swarm stack, that is why I’m using swarm_stack in the query.

I hope this quick step-by-step solution will help everyone who wants to setup the latest version of Grafana Loki in a Docker Swarm environment.