Log collector

After changing our architecture in order to improve average response time for our clients, we found a new problem: logs.

Before the change, all our logs were in a single machine, so accessing them was really easy. We had a rotate policy and our log files were not bigger than 50Mb. We found the log that we wanted with the text editor vim. (I know it is very “prehistoric”, but… you know, that was our first steps)

Now we have several machines with docker and a load balancer, and each machine is still using the same old way to log. So we have to go to each node and find the logs there. No way, it takes a lot of time. We have to change it.

We started googling “logging tools” and, after a little research, we discovered Scalyr. It has everything that we wanted, good pricing and it is made by ex-google developers, so… we choose it and we create the 30 days trial account.

How to use Scalyr

Configure Scalyr is very easy, because It is very well documented. Anyway, I’m going to show you our steps.

As I said, we have Docker in our architecture, so we started to follow the tutorial “install Scalyr agent with docker”. The idea is the next one: Scalyr is going to run in a new container and it is going to redirect every log that it receive to his cloud. Of course we also have to send the logs form the other containers to Scalyr container. In a conceptual way it is easy, but the docs have only examples about running docker in command line and no with the “new” docker-compose. So we have to “translate” everything to a docker-compose.

Hopefully for you, I’m going to share my docker-compose.yml file:

version: ‘2’
services:
  scalyr:
    image: custom-scalyr-image
    volumes:
      - /var/run/docker.sock:/var/scalyr/docker.sock
    ports:
      - “601:601”
postgres:
  [POSTGRES CONF]
  logging:
    driver: syslog
    options:
      syslog-address: “tcp://127.0.0.1:601”
wsgi:
  [WSGI CONF]
  logging:
    driver: syslog
    options:
      syslog-address: “tcp://127.0.0.1:601”
nginx:
  [NGINX CONF]
  logging:
    driver: syslog
    options:
      syslog-address: “tcp://127.0.0.1:601”

As you can see, we have to add the logging redirect to all the others containers that we have. The image custom-scalyr-image is build with this simply Dockerfile:

FROM scalyr/scalyr-docker-agent
COPY scalyr_api_key.json /etc/scalyr-agent-2/agent.d/api_key.json

In scalyr_api_key.json file, you have to have you api key, something like this:

{
  api_key: “MY_SCALYR_API_KEY”
}

(you can copy your api key to that file however you want, maybe you preferer sending it through env vars)

After redeploying everything, every log from the statement CMD from the Dockerfile will be recorded and saved to Scalyr. Furthermore, it also send data about CPU, RAM, network, etc. from the nodes where the containers are running.

EXTRA: More log files

Docker recommend that each container should just run one process, but sometimes we need to run more than one with tools like supervisord. In this case, maybe you like to log in different files, so you won’t send the logs that we need to stdout.

No worries, you can also recollect the logs from your files with Scalyr. In order to doing that, you have to install Scalyr agent in each container. I will follow the instructions from here. Realize that for this solution you won’t need the container with Scalyr, each container will send the logs to Scalyr cloud. So first of all, add this the Dockerfile where your logs are:

RUN wget -q https://www.scalyr.com/scalyr-repo/stable/latest/install-scalyr-agent-2.sh \
  -P /tmp/
RUN bash /tmp/install-scalyr-agent-2.sh \
  — set-api-key “MY_SCALYR_API_KEY” \
  — start-agent

Note: the script install-scalyr-agent-2.sh suppose that “python” command is “python2”, so you will have to change your path order to do that.

Finally, you have to say which files do you want to send to Scalyr. This is done through a config file in /etc/scalyr-agent-2/agent.json. You can add your own config file just doing this in your Dockerfile:

COPY scalyr.conf /etc/scalyr-agent-2/agent.json

Where scalyr.conf is something like this:

// Configuration for the Scalyr Agent. For help:
// https://www.scalyr.com/help/scalyr-agent-2
{
  api_key: “MY_SCALYR_API_KEY”,
  server_attributes: {},
  logs: [
    {
      path: “ROUTE_TO_LOG_FILE”,
      attributes: {parser: “DESIRED_PARSE”}
    },
  ],
  monitors: []
}

You can tunning your log configuration following this docs.

And that’s all! Enjoy your data! =)

Author: Jesús Sánchez (check me out)

Also check out Witei on Twitter, Facebook, or LinkedIn.

We are hiring! Send your magic to jobs@witei.com