Setting Up Sidekiq 6.0 on AWS

Soon after the release of Rails 6.0 and Vuetify 2.0, we at KITE decided it was time to perform the task of upgrading our dependencies; a daunting challenge for any developer. And so, I was charged with updating our Rails application which included the upgrade to Sidekiq 6.0.

Challenge accepted.

Now, I knew the Sidekiq upgrade was going to be a challenge. The deprecation of daemonization, logfile, and pidfile command line arguments meant a whole restructure of how our Sidekiq system was managed.

At the time, there weren’t a lot of resources on how to do this and available resources were misleading or out-of-date. So after banging my head against a wall for a few days, I figured it out and I’m excited to finally share this solution with the tech community!

Before we dive in

The tech stack we’re working with is an AWS Elastic Beanstalk (Puma with Ruby 2.6 running on 64bit Amazon Linux/2.12.1) running Rails and AWS ElastiCache(Redis 4.0+) which is necessary for Sidekiq.

My assumption is that you know how to SSH into your Elastic Beanstalk Application and your current Upstart version is `0.6.5`. I also assume you’ve deleted your previous Sidekiq config files OR are starting with a clean slate.

SIDE NOTE: You can find out your Upstart version by SSHing into your Elastic Beanstalk app and running the following command initctl version.

Let’s dive in

First, in the root of your project, create the following file config/sidekiq_worker.yml and set the queues for that worker.

:queues:
  - ["priority", 2]
  - ["default", 1]

This indicates that there will be two queues we can send jobs to: priority and default. For more information please visit Sidekiq’s Advanced Options Wiki page.

SIDE NOTE: If you want to run multiple Sidekiq processes, create another config file (i.e. config/sidekiq_worker2.yml).

Now we need to configure our Elastic Beanstalk application beyond what is available through the AWS console. This kind of advanced environment customization is accomplished with configuration files that are deployed with your application. You can follow this AWS guide on how to set up these advanced configuration files, but the quick and dirty is the following:

1. Create a folder named .ebextensions at the root of your application.
2. Files in this folder will run in lexicographical order
3. Files must conform to YAML or JSON specifications and have the .config file extension

These config files will run every time you deploy to your Elastic Beanstalk application.

Now that we know what .ebextensions is and how it works, let’s proceed in setting up our configuration files!

SIDE NOTE: In order to be 100% certain files will run in order, I like to use numbers to dictate the execution order of files. Please use whatever you feel comfortable with for naming conventions.

Configuration Files

In the .ebextensions folder create a new file 001_create_sidekiq_logs.config with the following content:

This will create a new .log file type in the directory /var/log during deployment. Notice the name of the log reflects the name of the .config file we created in the step before.

The next file we’re going to create will be the config file for our init process. Name the file something like 002_sidekiq_worker.config and add the following code.

Kudos to Mike Perham and the Sidekiq team! This is an adaption of their upstart sidekiq config file.

Sidekiq worker config for Upstart

Most of this stuff is straightforward; however, let’s focus on lines #36–37. As you can see, we are utilizing the config/sidekiq_worker.yml file to determine the queues available for this process. We then redirect STDOUT and STDERR to the /var/log/sidekiq_worker.log file we created in step 2.

SIDE NOTE: If you want multiple Sidekiq processes: duplicate the 002_sidekiq_worker.config file and rename it, make sure it executes BEFORE the file you’ll create in the next step (xxx_sidekiq_upstart.config ), change the name of the file that’s created, and change the STDOUT/STDERR output redirect to a different log you created from step 2. (i.e. … >/var/log/sidekiq_worker2.log 2>&1)

Pretty slick right?! The only issue is, Upstart doesn’t have access to this new process yet. Let’s make that happen!

In the .ebextensions folder, add a new file called 003_sidekiq_upstart.config with the following content:

Workers Upstart config to manage a set of Sidekiq processes

When this is executed, the first file created is a symlink to the sidekiq_worker.conf you created previously. Now, we could have placed the contents of 002_sidekiq_worker.config here, but for ease of file management this is a better solution.

The second file created, /etc/init/workers.conf, is another adaptation of the Sidekiq team’s general manager of Sidekiq processes on Upstart. Finally, a command is ran that reloads the Upstart configuration. This allows us to start and stop our new sidekiq_worker process!

If you want to test this, deploy your application to elastic beanstalk. SSH into your application and run sudo start sidekiq_worker. You can see your process running by using the command initctl list. VOALLA! MAGIC!

You’re probably thinking, “How do we automate this process?” Great question. KEEP READING!

Automating the Process

In order to mainstream this, we’re going to create two shell scripts; one that mutes our Sidekiq processes and one that restarts them.

Create a file called 004_sidekiq_mute.config in your .ebextensions folder.

Upstart 0.6.5 doesn’t provide an easy way for us to send a TSTP signal to quiet our workers. So, the work around for this is to do it ourselves. In the file we just created add the following code:

Mute Sidekiq processes shell script

This essentially finds the process id and sends a TSTP signal to it, effectively quieting the worker!

Now create a file called 005_sidekiq_restart.config in your .ebextensions folder and paste this code inside it:

Restart Sidekiq processes shell script

It basically does the same thing as the mute process, but sends TERM signals to the process followed by stopping and then starting the workers process we made in 003_sidekiq_upstart.config file.

The last thing to do is hook up the mute/restart shell scripts to be ran on the appropriate Elastic Beanstalk hooks!! Create ONE MORE FILE called 006_eb_sidekiq_hooks.config in .ebextensions with the following content:

Elastic Beanstalk Hooks for Sidekiq

Now deploy your application! Once it’s deployed you can confirm Sidekiq is working by SSHing into the server and running initctl list or you can hit Sidekiq’s GUI to make sure jobs are executing.

Congratulations!

THAT’S IT!! I hope you found this blog post helpful in setting up Sidekiq 6.0 on your Elastic Beanstalk application. Please reach out with any questions you may have and I’ll be happy to help!