Log Rotate: Managing Your Cluster Log Made Easy

Consider the following scenario, We have 10 remote machines which collectively make our cloud environment. We can have as many machines as we like. On these machines we have, say, frameworks and applications running. These applications also produce logs for monitoring applications to parse and visualize in the form of graphs for a better understanding of the data and the semantics associated with them are pretty clear with visualization tool like Kibana.

Let’s talk about monitoring tools, they rely on log files, that is, along with the help of certain system commands for system monitoring, these tools parse log files into formatted data. In terms of monitoring, we have to consider that we are not interested in historical data but what is currently going on with system/service(s). Now comes the big question.

Is it sensible to have large sized log files having data of previous year or month?

Some applications generate logs at a much faster rate. Which leads to log files having higher size. Now to solve this we can configure log rotation in Unix systems. But we have 10 systems to configure and if we are brave enough we can even more systems.

The answer is Log Rotation. We can easily configure log rotation in 10 or more systems with Ansible.

Logrotate Configuration File

We are going to take a look at a dummy log rotate file and let’s see what it means.

These configuration files can be found at this location var/logrotate.d/conf

What does __ mean, where __ is :

• rotate <count>: Log files are rotated count times before being removed. If the count is 0, old versions are removed rather than rotated.
• monthly: Log files are rotated the first time logrotate is run in a month
  (this is normally on the first day of the month).
• compress: Log files are rotated the first time logrotate is run in a month
  (this is normally on the first day of the month).
• missingok: If the log file is missing, go on to the next one without issuing an error message.
• notifempty: Do not rotate the log if it is empty.

This configuration file tells the logrotate job to pick the log from location /var/log/apt/term.log and ‘rotate’ them.

We will rely on Jinja templating for generating application-specific configuration files. Following is how our Jinja template looks:

What we are doing here is that for every definedlog_location in an environmentlocations variable. We will define the configuration for that application. Where the following environment variables stand for:

• period: defines the periodicity of rotation. It can be monthly, weekly or daily.
• rotate_count: defines the number of copies to be made of the log.
• size_threshold: sets the upper bound on the size of the log file to trigger log rotation.

There is one very important option called copytruncate . It works along the lines of ‘If it ain’t broke, don’t fix it’ which means that the log file will only be rotated when the log has new lines/data.

How will it know? Remember that even a letter written to a file can cause the size of the file to increase by a byte. That’s how it knows.

To see things in action on your machine/s make sure you have ansible(2.5.1 or later). Here’s the link to my Repo to view an example to apply logrotate on the logs of an application running over your cluster, make sure to go through the readme file.

Originally published at blog.knoldus.com on August 20, 2018.