Background jobs with less negative effects

Safe background job. By Grégory Tonon

Sometimes when we have a background processing system to run stuff that is not necessary in real time, we are tempted to use too much of it.

Since it runs concurrently, it is easy to overload the database with lots of heavy queries.

At Enjoei, we started to improve these background jobs (the “read only” jobs, of course) by making them reach our follower database.

Use cases

In our Rails stack we made the following enhancements:

Background Emails through Sidekiq

Since emails are not supposed to perform write operations on the DB, you can force them to run in background without hitting the main DB.

It will be necessary to set up Sidekiq to use a specific queue with a follower DB configured:

• Add the environment variable DATABASE_FOLLOWER_URL;
• Configure Sidekiq to send emails with the right queue;

# @ config/initializers/sidekiq.rb 
Sidekiq::Extensions::DelayedMailer.
 sidekiq_options queue: :read_only_default

• Run sidekiq pointing to the right database.

$ DATABASE_URL=$DATABASE_FOLLOWER_URL bundle exec sidekiq -q read_only_default

Extend read only option to other jobs

You can use a Sidekiq middleware to setup a worker that targets the follower database, like this:

class MyWorker
  include Sidekiq::Worker
  sidekiq_options db: :read_only
  def perform
    # ...
  end
end

Cron jobs

If you use Sidekiq Enterprise with periodic jobs, you can use the same middleware mentioned above, like we do.

We used to use Heroku Scheduler, the solution below applies if you are using something similar.

Rake tasks

Just set up the DATABASE_URL in the command:

$ DATABASE_URL=$DATABASE_FOLLOWER_URL rake reports:all

Summary

We think that the end user should not be negatively affected by background stuff. We are always trying to improve this by moving those runners outside of the “end user infrastructure.”

We have been applying those techniques for emails, push notifications, feeds and other indexations, reports, etc.

There are still lots of challenges, mainly when we have to write data, work with real time changes, and so on. We are constantly learning and applying new techniques and I will keep writing about it so we can exchange experiences even if some of them were not good.

In a future post, Anderson Dias will write about the Sidekiq middleware that we use, giving more details about the implementation.