Do not Fly Blind! Setup Grafana & Loki for Caprover (Docker Swarm)

Enhanced observability and monitoring for Docker workloads in Caprover ecosystem

Hi, In this article I am going to tell you how we can setup Grafana and Grafana Loki for our contanierized apps that are running on Caprover.

In this article, we are going to use Docker’s Loki plugin to send logs directly from containers to Loki instance.

Image by Loki

I firstly heard of blind flying term usage in IT from John Rofrano while tracking IBM DevOps and Software Engineering Professional Certificate program on Coursera. Day by day, I really understand why observing over than blind flying is so important. Hopefully, this article will help you setup your own fly observation center..

#1 Deploying Loki

I have created an app by using image grafana/loki:3.0.0 and named it loki on my Caprover which is exposing its port 3100.

I also set volumes by binding a path between the app and the host.

Obviously you should create an app by selecting the “has persistent data”.

Path binding

If you check the link below, you’ll see that they say you should pass an argument -config.file to set the loki-config.yaml . Since we can not pass this argument while using Caprover UI, we are going to copy the config file into the path that we bound for updating our loki service in order to use this file.

> cd /captain/data/nginx-shared/loki/config # we bound this path to the app's /mnt/config
> ls
loki-config.yaml

Install Loki with Docker or Docker Compose | Grafana Loki documentation

Now we can install Docker plugin to send app logs to our loki service.

docker plugin install grafana/loki-docker-driver:2.9.2 --alias loki --grant-all-permissions

And we are able to configure our apps’ configuration how they send their logs. At this point, we have 2 options:

• We can configure Docker daemon to send all Docker logs to our loki instance
• We can configure each services’ logging method to send their logs to our loki instance

I am going to apply the 2nd method, configuring the service that I need to track its logs on Grafana.

Before we configure the app logging mechanism, we should update our loki service to use the loki-config.yaml file.

docker service update \
    --args="-config.file=/mnt/config/loki-config.yaml" \
    srv-captain--loki # this is our service name

We updated our loki service to set the config.file because of In Docker Swarm, when updating a service with docker service update, passing additional arguments (like -config.file in this case) to the container's entrypoint or command can be done using the --args or --env flags, depending on how the container is designed to receive such parameters. For our use case, where we want to pass the -config.file argument to the grafana/loki:3.0.0 container, we can do it with the --args flag.

#2 Configuring Django App’s Logging

I have already deployed an instance of my donation-app project in my server.

GitHub - mebaysan/donation-app: A dream of developing a Software as a Service (SaaS) solution…

I am also going to update the app service for sending its logs to loki instance.

> docker service update \
    --log-driver loki \
    --log-opt loki-url=https://my-loki-domain/loki/api/v1/push \
    srv-captain--my-donation-app

# you should see a similar output
srv-captain--my-donation-app
overall progress: 1 out of 1 tasks 
1/1: running   [==================================================>] 
verify: Service srv-captain--my-donation-app converged

#3 Setting Up Grafana

Now, we are ready to observe and track the logs on our loki instance. Obviously, we are going to do that by using Grafana.

You can easily deploy a Grafana instance by using One Click Apps in Caprover.

After setting up grafana instance, we should add our loki instance to Grafana datasources.

Adding a datasource

Now we can go to Explore page and query our logs!

Image by Author

Finally

Hopefully, it is helpful to configure your own observability setup on Caprover.

You can find my links below to follow me on other platforms.

Kind regards

@mebaysan - IT Professional

BaysanSoft

mebaysan - Overview