Monitoring Cron Runtime In Ruby

Photo by Luca Bravo on Unsplash

At LocoNav, we have our web backend in Ruby on Rails, and we use many gems for different purposes. One such useful gem is whenever. We have thousands of cron tasks running per day. As our system scaled, it was essential to make sure these tasks run in the expected time. For e.g., if a task has to run every hour, we must make sure that it runs in less than an hour. If it takes more than the expected time, the relevant teams must get notified. To avoid multiple copies of cron, whenever provides a solution using linux file locks. Though whenever provided a very clean syntax for defining and deploying cronjobs, it does not provide any mechanism for tracking them. So we started building a simple in-house solution.

What We Wanted to Build

1. A method to calculate runtime of a cronjob.
2. A DSL to define the expected runtime of a job.
3. A datastore to persist the last runtime of a job.
4. A notification mechanism (Slack/ Hipchat/ Email).
5. A user interface to view the last runtime vs. expected runtime.

Here’s how we did it…

Calculating Cron Runtime

whenever provides two ways of running rails scripts: rake tasks and runners. One easy way to calculate runtimes is by using rake hooks. Rake provides start and end hooks which we can use to check the start time and end time of a cron job. So first, we moved all whenever crons to rake based scripts (we had few jobs which were written using runner syntax). However, these hooks run on all rake tasks (some of which may not be cron jobs), so we namespaced cron jobs properly to make this hook run for cron based rake tasks only. Also, we used Redis hashes to store start_time, end_time and task details.

Here is the sample code,

The after_logger notes end_time, takes the previous runtime from Redis and calculates job runtime. If this comes out to be more than the threshold, it can send a notification.

Syntax for Defining Cron Runtime

We checked whenever documentation and found we can define the following for a cronjob:

1. Running frequency (Eg. 1.day)
2. Time (Eg. 1:00 am)
3. Log files (standard and error)
4. Rake command details

The only thing we wanted to add was runtime_threshold. So for each task, we initially defined some decent threshold and modified it later as per actual job runtimes. We came up with our own hash based syntax in a module.

The SCHEDULES hash defines all the parameters we need. So now, the schedule.rb file looks like:

A Datastore to Keep Last Runtime for a Job

We are using redis to persist the runtime stats, code for which is wrapped in a module called CronMonitor. There is no particular need to use Redis here, and you can use MongoDB or a SQL database for storing them.

Here’s the sample code for redis based implementation:

A Notification Mechanism

Notifier.cron_threshold_exceeded(msg) in the code snippet is also kept as a black box. You can configure any notification mechanism in this class.

Here’s the sample code:

A User Interface to View Everything

We use active_admin as our admin platform. So, to build a view over this, we used active admin custom pages. We built a page to view this data and highlight the jobs which take longer than expected.

And then we emerged victorious! 💪

In the end, here’s how our UI looks like:

And this is the story of how we’re monitoring the runtime of cron jobs as the system is scaling!