Avoiding Unix’s logrotate with sed and tail

Or, My most egregious one-liner ever

Some of my cronjobs run on a shared hosting server where I don’t have access to /etc/logrotate or /var/log, so I can’t use logrotate to manage their logs.

They write log lines to files in a log directory in my home directory. All I’ll ever want from these logs is the most recent few lines anyway. To ensure those log files don’t grow without bounds, I wrote a one-liner cron job that keeps them within the 200 most recent lines.

This is the cron job command:

find $HOME/log -maxdepth 1 -name \*\.log -type f ! -executable -exec sh -c 'MAX=200; for f do if [ $(wc -l < "$f") -gt $MAX ]; then TMPF=$(mktemp) && tail -$MAX "$f" > $TMPF && chmod --reference="$f" $TMPF && mv $TMPF "$f"; fi done' sh {} +

How did I get to that unholy invocation? Let’s start with how to trim a file to its last 200 lines. At first I considered sed.

Using sed

Here’s a command that tells sed, “delete everything from the start to 200 lines from the bottom”. (That is, just leave the last 200 lines.)

To figure out “200 lines from the bottom”, we need to determine how many lines are in the file. “wc -l (file)” prints a string like “256 (file)” where the 256 is the number of lines. So we have to take only that field. We can do that with “cut -d ‘ ‘ -f 1”. Now that we have the string “256” we can turn it into a number and subtract 200 from it.

sed -i -e "1,$(expr $(wc -l log.txt | cut -d ' ' -f 1) - 200)d" log.txt

That command changes the file in place.

But sed had a problem: The command doesn’t work if the file doesn’t have enough lines. (200 lines in the above example.) So I considered the tail way.

Using tail

TMPF=$(mktemp) && tail -200 log.txt > $TMPF && mv $TMPF log.txt

The above code does the following:

1. Make a temp file.
2. Writes the last two hundred lines to the temp file.
3. Renames the temp file to the log file, overwriting it.

Doing the work on the side and changing the logfile in one atomic move is nice.

Putting it together

The above tail command makes a nice -exec predicate to a find command that’d list all the log files in the log directory:

find $HOME/log -maxdepth 1 -type f -name \*\.log -exec sh -c 'TMPF=$(mktemp) && tail -200 $1 > $TMPF && mv $TMPF $1' sh {} \;

The above command uses “find” to list all the files ending in .log in $HOME/log and run the tail command on them.

If you’re still with me, you’re probably wondering why the two “sh” commands. That’s to avoid a command injection vulnerability when using {} in sh -c.

The above invocation didn’t last long. At the time, I preferred “find | xargs” to “find -exec”, and I wanted to use the character % to represent the file. (In vim, % often represents the current file, so the association already exists.)

find $HOME/log -maxdepth 1 -type f -name \*\.log | xargs -I% sh -c 'TMPF=$(mktemp) && tail -200 % > $TMPF && mv $TMPF %'

After a little baking, the above morphed into something like the following:

find log -maxdepth 1 -name \*\.log -type f ! -executable -print0 | \
xargs -0 -I{} sh -c 'M=200; if [ $(wc -l < "{}") -gt $M ]; then \
TMPF=$(mktemp) && tail -$M "{}" > $TMPF && chmod --reference="{}" $TMPF && mv $TMPF "{}"; fi'

It finds all the non-executable files that end in “.log” in the log/ directory, and if they’re over 200 lines long, does the tail command, and assures the updated file retains the same permissions even after the move from the temp file.

Some considerations:

• It uses {} instead of %, because in cron jobs, “%” has to be escaped. (I like that % means “filename” to my vim-reading eyes, but {} can be used without escaping at both the command line and in cron.)
• I added -print0 and xargs -0 for better filename protection (names with spaces and single quotes).
• Quoting {} allows for filenames with spaces and single quotes, but still allows for command injection with double-quotes, so we’ll have to abandon {} after all.
• Put the work inside a conditional statement so that logfiles don’t get touched if they’re not too long already. (I could have also ran “touch -r sourcefile targetfile” to restore the timestamps, but that seems more like lying.)
• I added ! -executable just for another sanity check on the file permissions.
• I put the -name check before -type in order to avoid having to call stat(2) on every file.
• Remember how I had “wc -l log.txt | cut -d ‘ ‘ -f 1” to cut off the filename from the wc output? I learned I could get the linecount only by having wc read from stdin instead. Eg., “wc -l < log.txt”.

Replacing the {} arguments with positional arguments like %1 gets us closer to the one-liner “unholy invocation” we saw at the top that serves as a cronjob:

find $HOME/log -maxdepth 1 -name \*\.log -type f ! -executable -print0 | xargs -0 -I{} sh -c 'MAX=200; if [ $(wc -l < "$1") -gt $MAX ]; then TMPF=$(mktemp) && tail -$MAX "$1" > $TMPF && chmod --reference="$1" $TMPF && mv $TMPF "$1"; fi' sh {} \;

There’s one more improvement to make! Instead of opening a shell for each filename found, they can all be passed into one shell that’ll iterate over them. (But to do so, don’t use “find | xargs”, go back to “find -exec”. We’ll be able to drop the -print0 too.)

find $HOME/log -maxdepth 1 -name \*\.log -type f ! -executable -exec sh -c 'MAX=200; for f do if [ $(wc -l < "$f") -gt $MAX ]; then TMPF=$(mktemp) && tail -$MAX "$f" > $TMPF && chmod --reference="$f" $TMPF && mv $TMPF "$f"; fi done' sh {} +

There we go. Filenames with spaces and quotes are supported, only one shell is made for the command, and logs are trimmed responsibly, keeping their permissions and being done in a relatively atomic way.