Rails background workers: Sidekiq overview and how to deploy it to Heroku

Sidekiq is a framework for background job processing that is very useful for handling expensive computation, emails, and other processes that is better served outside of the main web application.

In this blog post, I use a rather quirky example of parsing a rather large database of Pokemons as a proxy in explaining the need for Sidekiq, the workflow of Sidekiq, how to implement/leverage Sidekiq, and finally how to deploy Sidekiq to Heroku.

A contrived but helpful example of when you need Sidekiq

Feel free to follow along to this guide by downloading this pokedex repo here, or play around with the deployed app here.

Suppose you found an awesome CSV file listing all 807 Pokemons and you wanted to create a Rails app where you press a button and it’d create 807 instances of a Pokemon class and display them in a table format like so:

So now you may be tempted to write up something like this in our PokemonsController

#app/controllers/pokemons_controller.rb
class PokemonsController < ApplicationController
require 'csv'
  def index
@pokemons = Pokemon.all
end
  def upload
csv_path = File.join Rails.root, 'db', 'pokemon.csv'
CSV.foreach((csv_path), headers: true) do |pokemon|
Pokemon.create(
pokedex_id: pokemon[0],
name: pokemon[1],
height: pokemon[3],
weight: pokemon[4])
end
flash[:notice] = "All 807 Pokemons added to db."
redirect_to pokemons_path
end
  def destroy_all
Pokemon.destroy_all
flash[:notice] = "All 807 Pokemons deleted from db."
redirect_to pokemons_path
end
end

While this code is functional, once the user hits either the “Import Pokemons” or the “Delete Pokemons” button, it will lock up the browser until every single 807 instances of Pokemons gets created or deleted since it is now processing the same parsing/creating/deleting action in the same thread as the request/response.

This is annoying in development mode, but say this were to happen to your deployed Heroku app, you may even end up with something like this:

This is where Sidekiq comes in. Instead of having the main web app thread handle the creation/deletion of Pokemons, you can instead punt this tedious and memory-consuming task over to your trusty Sidekiq — a background worker that allows you to execute tasks outside of the main web application thread. Take this beautiful chart for visualization purposes:

Courtesy of https://www.youtube.com/watch?v=GBEDvF1_8B8

Instead of the controller handing off all the creation/deletion of Pokemons in the Pokemon model class, then sending the results back to the web server and to the computer screen, the goal here is to delegate this task out to a Redis queue (more on that later) and have Sidekiq tackle these tasks while the web app thread gently lets the users know that creating/deleting all Pokemons are currently being worked on.

The Setup

First, add Sidekiq to the Gemfile and bundle

gem 'sidekiq'

Sidekiq relies on Redis — a database store which in this case will be used as a way to hold the Pokemon creating/deleting jobs in a queue. To install locally, type in Terminal:

brew install redis

Refactoring our Pokemon code to leverage Sidekiq

After our Sidekiq gem is installed, you are now able to create a couple Worker classes to handle the addition and deletion of Pokemons like so:

#app/workers/pokemon_worker.rb
class PokemonAddWorker
include Sidekiq::Worker
sidekiq_options retry: false
  require 'csv'
  def perform(csv_path)
CSV.foreach((csv_path), headers: true) do |pokemon|
Pokemon.create(
pokedex_id: pokemon[0],
name: pokemon[1],
height: pokemon[3],
weight: pokemon[4]
)
end
end
end
class PokemonRemoveWorker
include Sidekiq::Worker
sidekiq_options retry: false
  def perform
Pokemon.destroy_all
end
end

As you can see above, to turn a plain Ruby class to a Sidekiq Worker class, you just need to add in the line include Sidekiq::Worker below the class declaration.

Additionally, by default, Sidekiq will continuosly retry methods that fail. There are cases when this wouldn’t be ideal, such as whenever credit cards are involved. If so, you can turn it off like in the code above by adding in: sidekiq_options retry:false

After adding those lines in, add in a #perform method, and now you can just drop in any tasks inside the method that you don’t want to bottleneck the main web-app with.

Now you just need to refactor the PokemonsController.rb by calling on these Worker methods.

#app/controllers/pokemons_controller.rb
class PokemonsController < ApplicationController
  def index
@pokemons = Pokemon.all
end
  def upload
csv_path = File.join Rails.root, 'db', 'pokemon.csv'
PokemonAddWorker.perform_async(csv_path)
flash[:notice] = "Pokemons getting added to db"
redirect_to pokemons_path
end
  def destroy_all
PokemonRemoveWorker.perform_async
flash[:notice] = "Pokemons getting removed from db"
redirect_to pokemons_path
end
end

Notice that the Worker methods are executed by just writing out NameOfWorkerClass.perform_async(args)

Executing everything (at least locally)

You will need three tabs of Terminal open for this:

Tab 1: Fire up a redis-server by typing in redis-server

Tab 2: Fire up your Sidekiq by typing in bundle exec sidekiq

Tab 3: Fire up your actual Rails app by typing in rails s

If you were to go to the localhost:3000 in the browser, you will be able to see that your Sidekiq is working and not locking up your main browser when importing or deleting your Pokemon database.

Instead of locking up the screen until all ~800 Pokemons get loaded to the database, Sidekiq lets us continously refresh the page to see how many Pokemons have gotten created.

Deploying Sidekiq to Heroku

So you may now be wondering: “I can’t open up several Terminals in the Heroku cloud and brew install redis server and redis-server and bundle exec sidekiq right? Right??

Fortunately, Heroku makes all of this pretty streamlined.

Getting Redis to Heroku

Once the main Rails app is deployed to Heroku, log in to your Heroku account then go to the app’s dashboard and click the Resources tab.

Scroll down to the Add-ons section and search for Redis To Go, and install the free version of it. At the end of it all, it should look something like this:

Now go to the Settings tab on the Heroku app dashboard, and click Reveal Config Vars. Add in a Key-Value pair of REDIS_PROVIDER and REDISTOGO_URL respectively at the end of the list. It should look like this if you did it correctly:

This in turn is the local environment equivalent of brew install redis and redis-server

Getting Sidekiq to Heroku

If you haven’t already, create a Procfile in the root of your Rails app. Straight from the Heroku docs:

A Procfile is a mechanism for declaring what commands are run by your application’s dynos on the Heroku platform. It follows the process model. You can use a Procfile to declare various process types, such as your web server, multiple types of workers, a singleton process like a clock, or the tasks you would like to run before a new release is deployed to production.

If you’ve ever deployed a Rails app to Heroku without a Procfile, you may have noticed a warning in the logs stating that no Procfile was detected and that a default value for the app was set as:

web: bin/rails server -p $PORT -e $RAILS_ENV

This is just Heroku being really nice, but it won’t be nice enough to sense that you’re using Sidekiq and add in proper Procfile commands for you.

Instead, go into your Procfile and enter in:

pokemonworker: bundle exec sidekiq -c 2

Critical Aside on -c 2 flag

Sidekiq prides itself on being able to run several threads concurrently (what the ‘c’ stands for in the -c 2 flag). It’s similar to this amazing guy at the left just killing it at an arcade basketball game:

Guy on Left: bundle exec sidekiq -c 2; Guys on Right: bundle exec sidekiq -c 1

By default, bundle exec sidekiq will actually run 25 threads concurrently, but you have to realize that that’s a lot of threads, and that we are using a free version of Heroku on top of a free version of Redis To Go. So to avoid any Heroku logs that look something like:

2018-04-23T07:12:09.453785+00:00 app[pokemonworker.1]: 4 TID-qkxr8 WARN: Redis::CommandError: ERR max number of clients reached

we will just settle with two threads instead of 25. Hence we append our bundle exec sidekiq with a c -2 flag.

After pushing the code to Heroku, go back to the Resources Tab on the Heroku app dashboard and turn on the pokemonworker under the dyno section. When all is said and done, it should look like this:

The picture below shows Heroku logs (to display the logs, type in Terminal: heroku logs -t) of the app that’s deployed here. The logs highlighted in blue are logs created by the main web-app thread, while the logs higlighted in green are logs created by the Sidekiq/Redis threads. You can see that both are working concurrently.

So that should be it! Your Sidekiq is now deployed alongside your actual Pokemon creating/deleting app.

Credits/Further Resources

As with my previous blog, this blog would not have been possible without several excellent resources. I will list the main ones here, and I highly recommend that you check them out to get an in-depth understanding of Sidekiq and background processing.

Long-Running Tasks In Rails by Flatiron School: A succint rundown on how to set up a local environment of Sidekiq and how to implement it.

Background Processing with Rails, Redis and Sidekiq by Decypher Media: A 15 minute YouTube video jam-packed with high-level concepts on background processing (flowchart at top of blog post is from there) along with a great walkthrough on how to deploy Sidekiq to Heroku.

Sidekiq Playlist by DailyDrip: An official walkthrough by the creators of Sidekiq, it includes a great demo on displaying how Sidekiq works in parallel/concurrent threads along with showing in-depth tutorial of how to use its Sinatra based web-ui for seeing under the hood of Sidekiq operations.

Let’s Chat by Garrett Heinlen of Netflix: A talk given at a FogCity Meetup, gave quick overview of Redis, and how Sidekiq communicates with Redis.

Closing Thoughts

Thanks for making it all the way to the end, this blog turned out to be a lot more all-encompassing than I had envisioned.

Feel free to follow me on Twitter @kevinyckim. I make sure to tweet daily.