Setting up Sidekiq, Redis on AWS Elastic Beanstalk with ElastiCache

Sidekiq has been my favorite tool for setting up background processing in Ruby on Rails applications. Mainly I use it for delivering emails, and it can take a little effort to get up and running for the first time in the AWS ecosystem.

Redis is required for sidekiq, and AWS ElastiCache redis clusters pair nicely if you are already using Elastic Beanstalk. Here is my quick rundown for getting the sidekiq & redis configured.

First add the gems to your gemfile & bundle install. You may also need to run `gem install redis` on your system.

gem 'sidekiq'
gem 'sinatra', require: false #used for web interface

In your routes.rb file mount the sidekiq web interface. I use Devise, so here we’re wrapping the route in an authentication block which is smart for production. The authentication block can be omitted, or browse these for your authentication solution. In this example, the user class responds to an admin method which verifies the user is an admin.

require ‘sidekiq/web’
authenticate :user, lambda { |u| u.admin } do
mount Sidekiq::Web => ‘/sidekiq’
end

At this point I’ll mention the devise-async gem which allows you to send Devise emails asynchronously. First install the gem. Then you will need to add :async to your user model’s devise call.


# /app/models/user.rb
...
devise :database_authenticatable, :async, :registerable, ...

Then devise-async can be configure to use sidekiq by adding this short configuration file in /config/initializers/devise_async.rb

Devise::Async.backend = :sidekiq

Active Job will also need to be configured in your application to use sidekiq with Action Mailer. In /config/application.rb you can add the setting:

config.active_job.queue_adapter = :sidekiq

We will now need a configuration file for sidekiq created in /config/initializers/sidekiq.rb

rails_root = Rails.root || File.dirname(__FILE__) + ‘/../..’
rails_env = Rails.env || ‘development’
redis_config = YAML.load_file(rails_root.to_s + ‘/config/redis.yml’)
redis_config.merge! redis_config.fetch(Rails.env, {})
redis_config.symbolize_keys!
Sidekiq.configure_server do |config|
config.redis = { url: “redis://#{redis_config[:host]}:#{redis_config[:port]}/12” }
end
Sidekiq.configure_client do |config|
config.redis = { url: “redis://#{redis_config[:host]}:#{redis_config[:port]}/12” }
end

You’ll see we are loading some redis configuration in sidekiq.rb. Create this file in config/redis.yml and add the following:

development:
host: ‘localhost’
port: ‘6379’
test:
host: ‘localhost’
port: ‘6379’
production:
host: ‘your.redis.node.will.go.here.usw2.cache.amazonaws.com’
port: ‘6379’

Sidekiq can take an optional yml config file, which I found necessary for specifying which queues to have it watch. By default, sidekiq only watches a ‘default’ queue. Devise will add emails to a ‘mailer’ queue, and my mailer class adds messages to a ‘mailers’ queue. So let’s add each to the config file in /config/sidekiq.yml. The concurrency setting is optional, default for sidekiq is 25.

:concurrency: 5
:queues:
— mailer
— mailers
- default

At this point everything should be set up to work in your development environment. To send emails asynchronously make sure you use the deliver_later call on you messages. Ex: ApplicationMailer.new_registration(user).deliver_later. Then to start sidekiq run `rails s`, `redis-server`, and `bundle exec sidekiq` in terminal tabs.

Now our application configuration is pretty much complete, but we have to get redis working in our production environment. From your AWS dashboard select ElastiCache under the database heading. Click ‘Launch Cache Cluster’ and choose Redis as the engine.

The default options all work, and you can specify if you want to create a replication group in case of failover. The current smallest node type (cache.t2.micro) seems to work fine for small to medium applications.

In the advanced settings you can select a VPC security group. Use the default selection — default(vpc-xxxxxxx). After saving your new cluster, it will take a few minutes to provision. When it completes, from the Cache Clusters dashboard click the link under Nodes for your new cluster. It will have an endpoint name like my-redis-group.abcdef.usw2.cache.amazonaws.com Copy that endpoint name into the production host value in /config/redis.yml.

Now that VPC group’s security settings need to be updated to accept incoming connections. Open your EC2 Dashboard in the AWS console, and select Security Groups under Network & Security. Find the ‘default’ group that was created and select the Inbound tab. Click Edit and add a Custom TCP rule with 6379 as the port range and ‘Anywhere’ as the source. This guide from AWS can be helpful for this step. When this is complete, your Elastic Beanstalk applications created in the same account will be able to connect to your redis cluster.

The final step is creating the deployment commands so sidekiq is started/shut down when your application is deployed. You can find a number of gists with helpful scripts for this step. This is what I got to work with my app on a 64bit Amazon Linux 2015.09 v2.0.4 running Ruby 2.2 (Puma).

Add 0002_sidekiq_initial.config & 0003_sidekiq_setup.config to your .ebextentions folder:

Update 6/29/2016 — thanks Jeff Deville for the updated 0002_sidekiq_initial.config script which restarts sidekiq on environment variable changes and manual app restarts.

Update 10/24/2016 — (Heads up! It seems using variables like RESTART_SIDEKIQ can now cause issues in Elastic beanstalk…skip these next two files and read more below)

Update 10/24/2016 continued — On a new project I just had to copy the contents of RESTART_SIDEKIQ and MUTE_SIDEKIQ under each file. I ended up creating three total config files for sidekiq:

Now when you eb deploy (or change environment variables / restart app server), your instances will start up and shut down sidekiq gracefully.