Sidekiq is an efficient way to process background jobs in ruby. Sidekiq makes use of threads to process multiple jobs at same time in the same process.

By default, a sidekiq process creates 25 threads. Though the amount of concurrency can be tuned by changing the concurrency in sidekiq configuration file.

Some of the pertinent questions related to sidekiq scaling :

  1. What is that optimum number of concurrency for a sidekiq process?
  2. Does Increasing concurrency leads to increasing performance of sidekiq?
  3. How we can get maximum throughput from sidekiq?

Here, in this blog, I would mainly discuss about the issues that I faced while handling at least a million jobs using sidekiq and how I was able to improve the performance. (This was during my short stint in a well reputed tech based company. )

The incumbent Set up :

Server : A single core system with 8 Gb RAM

Requirement : 10 lakhs jobs to be processed.

Duration: It took around 17–18 hours to process 10 lakhs jobs

Sidekiq: The concurrency of sidekiq process was 100

Assumption: Increasing sidekiq concurrency would increase the throughput.

Processing a million jobs in 17–18 hours, clearly shows that there were serious problems with the existing architecture, code or configurations. The dev ops increased the concurrency till they started facing Database connection issues.

So what should be the max concurrency of a sidekiq worker ?

“Don’t set the concurrency higher than 50. I’ve seen stability issues with concurrency of 100, for example. Note that ActiveRecord has a connection pool which needs to be properly configured in config/database.yml to work well with heavy concurrency. Set the pool setting to something close or equal to the number of threads” - Mike Perham, Sidekiq author

Then how should we increasing sidekiq’s performance?

The first thing I experimented was to see the impact of increasing the CPUs on performance. The server configuration was changed to dual core. It had no impact for the following reason :

‘Each Sidekiq process running on MRI will only use one core, regardless of the number of threads. To get the benefit of multiple cores, you should run several Sidekiq processes.’ — Sidekiq help

Conclusion : To run multiple sidekiq workers to make use of all the available cores.

New System Configuration:

Server : Dual Core with 8 Gb RAM

Requirement : 10 lakhs jobs to be processed.

Sidekiq : 3 sidekiq workers with configuration of 50 each.

Results : It took around 3–4 hours to process a million jobs.

The change in set up resulted in 4–5 times better performance. This was a significant improvement with scope to further improve it by tweaking the concurrency number, sidekiq workers and multiple cores.

Attaching a response to this post from the Sidekiq Author himself, Mike Perham

PS: It would be great if someone can share a way through which it can be determined the opt number for sidekiq concurrency , sidekiq workers and multiple cores.

Co-founder at Knudge.me. Blog about ed-tech, growth, product, tech and more.

Co-founder at Knudge.me. Blog about ed-tech, growth, product, tech and more.