Moving from heroku scheduler to clockwork and sidekiq

We recently moved away from heroku scheduler and start using clockwork and with ActiveJob. The added value of such a move is massive:

• Better way to track changes in the scheduler
• Easier to get errors from failing tasks
• We could run any task at any time and with any interval

We wanted to configure clockwork in a way we could:

• Avoid executing a tasks before the previous run completed
• Catch any exception during a task execution
• We didn’t want to loose the ability to run tasks manually on heroku

The way we setup the project is pretty interesting and this is what I’d like to share with you, my amazing reader.

Clockwork

Just add the gem to your project

gem 'clockwork'

bundle and create a scheduler.rb file on the root directory for you project. Our scheduler.rb file looks like this:

# scheduler.rb

require File.expand_path('../config/boot',        __FILE__)  
require File.expand_path('../config/environment', __FILE__)  
require 'clockwork'

module Clockwork  
  every(10.minutes, 'Carwow: Recalculate stats') do
    ScheduledTasks::Carwow::RecalculateStats.perform_unique_later
  end

  error_handler do |error|
    Bugsnag.notify(error)
  end
end

This will execute the Carwow::RecalculateStats active job task every 10 minutes. The error handler will notify us in case we’ve got any problem in the scheduled task invocation (e.g. if there’s a typo in the task name)

Active job task

This is how our RecalculateStats job looks like:

# app/jobs/scheduled_tasks/carwow/recalculate_stats.rb

module ScheduledTasks  
  module Carwow
    class RecalculateStats < ScheduledTask
      def perform
        StatsGenerator.new.recalculate
      end
    end
  end
end

There’s some magic happening in the ScheduledTask class:

# app/jobs/scheduled_tasks/scheduled_task.rb

module ScheduledTasks  
  class ScheduledTask < ActiveJob::Base
    queue_as :scheduled_tasks

    rescue_from(Exception) do |e|
      Bugsnag.notify(e)
    end

    def self.perform_unique_later(*args)
      if self.task_already_scheduled?
        logger.warn "Task #{self.to_s} already enqueued/running."
        return
      end

      self.perform_later(*args)
    end

    def self.task_already_scheduled?
      job_type = self.to_s
      queue_name = 'quotes_site_scheduled_tasks'
      q = Sidekiq::Queue.new(queue_name)
      is_enqueued = q.any? { |j| j['args'][0]['job_class'] == job_type }

      workers = Sidekiq::Workers.new
      is_running = workers.any? do |x, y, work| 
        work['queue'] == queue_name && 
        work['payload']['args'][0]['job_class'] == job_type
      end

      is_enqueued || is_running
    end
  end
end

perform_unique_later checks against our sidekiq queue if the job is already there, this is to prevent long running jobs to be executed multiple times and start overlapping.

As you can see we also catch any exception and we send it to Bugsnag, look into it. This also prevents the task to be executed again and to stop retrying failing tasks.

What’s missing?

To run the scheduled task manually on heroku, and to test them on our local machine we created some rake tasks dinamically based on the tasks defined above:

# lib/tasks/scheduled_tasks.rake

namespace :scheduled_tasks do  
  require "./app/jobs/scheduled_tasks/scheduled_task"
  Dir[File.join('.', "app/jobs/scheduled_tasks/**/*.rb")].each{ |f| require f }

  classes = ObjectSpace.each_object(Class).select{|klass| klass < ScheduledTasks::ScheduledTask }

  classes.each do |klass|
    class_name = klass.to_s
    task_name = class_name.gsub("ScheduledTasks::", "").gsub("::", ":").underscore

    desc "Runs #{class_name}"
    task task_name => :environment do
      puts "Executing job #{klass.to_s}"
      klass.perform_now
    end
  end
end

With this bit of code we can now execute the previously defined rake task invoking

$ rake scheduled_tasks:carwow:recalculate_stats

That’s all.

Originally published at underthehood.carwow.co.uk on January 8, 2016.

Interested in making an Impact? Join the carwow-team!
Feeling social? Connect with us on Twitter and LinkedIn :-)