Sidekiq: to pay or not to pay
Not so long ago at Cheetah, we decided to use some cool features from the Sidekiq environment.
The first things we wanted to add were: batches, unique jobs and cron jobs.
We were already using open source versions of unique jobs and cron jobs, but when we decided to use batches, my first thought was to buy a paid license for Sidekiq Enterprise. It sounded really good, and there were tons of useful features!
- Encryption
- Periodic Jobs
- Unique Jobs
- Batches
- Multi-Core Processing
- And more…
It looked like even more than we needed, and everything was out of the box. The dream!
But when we tried to migrate existing code to new tools, there were some surprises.
Let’s discuss the top 3.
Cron Jobs
This feature has a lot of names: cron jobs, periodic jobs, etc.
But it doesn’t matter. We are using sidekiq-cron gem:
And there weren’t any issues with it. It was very stable and we didn’t need to take precautions with it. However, we are using TimeZone-specified crontab expressions like:
It’s a very useful feature because you shouldn’t care about re-calculating time for a specific area where your app should fire a cron job. Under the hood, timezone is used during the process of setting next occurrence time.
Here is a good description of how cron jobs work:
We got completely used to using such notation which helps to see the real picture when you have tons of different cron jobs in your system.
Unfortunately, Sidekiq Enterprise doesn’t have such functionality. There is only one way is to calculate all expressions and set them inside the file. But on other hand there is DST issue. Different countries change their time on different dates and it becomes very complicated to handle. Even if you calculate all expressions with corrections to your app’s TimeZone (e.g. UTC) you will be surprised with the fact that one day some of your cron will fire 1 hour earlier or later.
If you have read about how cron works, you can see that there is nothing too special in cron jobs itself, and there are not any killer features. But for our team, TimeZone-specified notation really helps to prevent any headaches.
So, possibly we haven’t faced any global issues which might be prevented by using Sidekiq Enterprise Periodic Jobs in our production yet, but for now I don’t see any reason to pay for it when free open-source tools do the same and even better 😋
Unique Jobs
Another must-have feature is unique jobs. Long story short — it prevents the job from being performed twice with the same arguments at the same time.
We are using this open-source gem:
It’s really good. There are different strategies for handling cases when your job is not unique and almost every case can be tuned as needed. Also you can choose how to unlock the job. It can be unlocked after a job is executed, when execution starts, or after TTL (Time To Live) expires.
On the other hand, there is built-in feature inside Sidekiq Enterprise which handles the same things. However, this is a potential bottleneck in my opinion. It uses TTL as main factor for unlocking. Even if the job wasn’t executed yet, but TTL is over — the job is unlocked. It’s possible to calculate TTL for every job approximately, but there is a chance to duplicate a job when something did not go well, and some jobs are running a bit longer than you thought.
Almost always I really don’t want to calculate TTL for the job and the option to lock it until it’s finished is good enough.
Lastly, another small feature is the list of unique digest in the Sidekiq admin area which is inside an open-source tool, but the paid one doesn’t have it. It doesn’t need to be present, but sometimes it’s very useful to see how many locks you have right now. And you can remove the lock with one click from that page, which looks nice!
So, in my case, the open-source version is better.
Batches
The last thing we are going to use is batches. It’s an amazing tool inside complex systems. This gives you the ability to group the jobs and make them synchronized.
It’s a part of Sidekiq Enterprise, but also there is open-source version:
I tried to compare which is better, but I can’t see any difference currently. They even have the same API. That’s why docs and guides for the paid version are actual for free one.
For this tool, I can say that they are equal.
Conclusion
Of course, this is all my personal opinion and based on experience I got while working on specific tasks. In other cases this may work differently.
But, I think that there is no reason to buy the Sidekiq license while we have such amazing open-source alternatives. They might have some issues or bugs, but they have a huge community and you don’t need to rely on customer support of the licensed product. And it’s a great resource for the open-source community and people who are happy to share their work with us.
