Setting Up Sidekiq 6.0 on AWS
Soon after the release of Rails 6.0 and Vuetify 2.0, we at KITE decided it was time to perform the task of upgrading our dependencies; a daunting challenge for any developer. And so, I was charged with updating our Rails application which included the upgrade to Sidekiq 6.0.
Challenge accepted.
Now, I knew the Sidekiq upgrade was going to be a challenge. The deprecation of daemonization, logfile, and pidfile command line arguments meant a whole restructure of how our Sidekiq system was managed.
At the time, there weren’t a lot of resources on how to do this and available resources were misleading or out-of-date. So after banging my head against a wall for a few days, I figured it out and I’m excited to finally share this solution with the tech community!
Before we dive in
The tech stack we’re working with is an AWS Elastic Beanstalk (Puma with Ruby 2.6 running on 64bit Amazon Linux/2.12.1) running Rails and AWS ElastiCache(Redis 4.0+) which is necessary for Sidekiq.
My assumption is that you know how to SSH into your Elastic Beanstalk Application and your current Upstart version is `0.6.5`. I also assume you’ve deleted your previous Sidekiq config files OR are starting with a clean slate.
SIDE NOTE: You can find out your Upstart version by SSHing into your Elastic Beanstalk app and running the following command
initctl version
.
Let’s dive in
First, in the root of your project, create the following file config/sidekiq_worker.yml
and set the queues for that worker.
:queues:
- ["priority", 2]
- ["default", 1]
This indicates that there will be two queues we can send jobs to: priority and default. For more information please visit Sidekiq’s Advanced Options Wiki page.
SIDE NOTE: If you want to run multiple Sidekiq processes, create another config file (i.e.
config/sidekiq_worker2.yml
).
Now we need to configure our Elastic Beanstalk application beyond what is available through the AWS console. This kind of advanced environment customization is accomplished with configuration files that are deployed with your application. You can follow this AWS guide on how to set up these advanced configuration files, but the quick and dirty is the following:
- Create a folder named
.ebextensions
at the root of your application. - Files in this folder will run in lexicographical order
- Files must conform to YAML or JSON specifications and have the
.config
file extension
These config files will run every time you deploy to your Elastic Beanstalk application.
Now that we know what .ebextensions
is and how it works, let’s proceed in setting up our configuration files!
SIDE NOTE: In order to be 100% certain files will run in order, I like to use numbers to dictate the execution order of files. Please use whatever you feel comfortable with for naming conventions.
Configuration Files
In the .ebextensions
folder create a new file 001_create_sidekiq_logs.config
with the following content:
This will create a new .log
file type in the directory /var/log
during deployment. Notice the name of the log reflects the name of the .config
file we created in the step before.
The next file we’re going to create will be the config file for our init process. Name the file something like 002_sidekiq_worker.config
and add the following code.
Kudos to Mike Perham and the Sidekiq team! This is an adaption of their upstart sidekiq config file.
Most of this stuff is straightforward; however, let’s focus on lines #36–37. As you can see, we are utilizing the config/sidekiq_worker.yml
file to determine the queues available for this process. We then redirect STDOUT
and STDERR
to the /var/log/sidekiq_worker.log
file we created in step 2.
SIDE NOTE: If you want multiple Sidekiq processes: duplicate the
002_sidekiq_worker.config
file and rename it, make sure it executes BEFORE the file you’ll create in the next step (xxx_sidekiq_upstart.config
), change the name of the file that’s created, and change theSTDOUT
/STDERR
output redirect to a different log you created from step 2. (i.e.… >/var/log/sidekiq_worker2.log 2>&1
)
Pretty slick right?! The only issue is, Upstart doesn’t have access to this new process yet. Let’s make that happen!
In the .ebextensions
folder, add a new file called 003_sidekiq_upstart.config
with the following content:
When this is executed, the first file created is a symlink to the sidekiq_worker.conf
you created previously. Now, we could have placed the contents of 002_sidekiq_worker.config
here, but for ease of file management this is a better solution.
The second file created, /etc/init/workers.conf
, is another adaptation of the Sidekiq team’s general manager of Sidekiq processes on Upstart. Finally, a command is ran that reloads the Upstart configuration. This allows us to start and stop our new sidekiq_worker process!
If you want to test this, deploy your application to elastic beanstalk. SSH into your application and run sudo start sidekiq_worker
. You can see your process running by using the command initctl list
. VOALLA! MAGIC!
You’re probably thinking, “How do we automate this process?” Great question. KEEP READING!
Automating the Process
In order to mainstream this, we’re going to create two shell scripts; one that mutes our Sidekiq processes and one that restarts them.
Create a file called 004_sidekiq_mute.config
in your .ebextensions
folder.
Upstart 0.6.5 doesn’t provide an easy way for us to send a TSTP signal to quiet our workers. So, the work around for this is to do it ourselves. In the file we just created add the following code:
This essentially finds the process id and sends a TSTP signal to it, effectively quieting the worker!
Now create a file called 005_sidekiq_restart.config
in your .ebextensions
folder and paste this code inside it:
It basically does the same thing as the mute process, but sends TERM signals to the process followed by stopping and then starting the workers
process we made in 003_sidekiq_upstart.config
file.
The last thing to do is hook up the mute/restart shell scripts to be ran on the appropriate Elastic Beanstalk hooks!! Create ONE MORE FILE called 006_eb_sidekiq_hooks.config
in .ebextensions
with the following content:
Now deploy your application! Once it’s deployed you can confirm Sidekiq is working by SSHing into the server and running initctl list
or you can hit Sidekiq’s GUI to make sure jobs are executing.
Congratulations!
THAT’S IT!! I hope you found this blog post helpful in setting up Sidekiq 6.0 on your Elastic Beanstalk application. Please reach out with any questions you may have and I’ll be happy to help!