You are not an Async(Sidekiq) professional if you don’t know these tricks

Lots of people miss essential settings or use Sidekiq in a poor efficiency way. 9 tricks make you Sidekiq fly.

I am preparing for the IELTS exam, so I’m trying to write articles in English. If you find any grammatical errors or technical issues, please do not hesitate to correct me in the reply section below.

Do not try to delete jobs

You must have lots of jobs you don’t need anymore which you want to delete, but it indeed inefficiency if you try to delete.

In my case, there were more than a million jobs in my process, so query a job just like looking for a needle in a haystack.

Even though you might think your case not large quantity like me, or you might think Redis is doing so fast or Sidekiq Pro version could solve.

Here a trick could efficiently resolve this issue.

Create a xxx_jid column on the table for saving jid which job you want to perform. Code ‘If’ clause to execute if current provider_job_jid equal what jid you saved, otherwise return the method.

https://gist.github.com/lustan3216/928fa11a6f23c24ed0eb748c722caba1

The context is an advertisement which can be resumed or paused, and both actions can be scheduled in the future.

Bugs will produce a situation which you want to change actions multiple times.

Why?

If you set pause job at a month later, later change to two months.

The bug will happen when the first schedule you set, and it will go first, then this situation could be complicated as the resume and pause action do multi times.

So why don’t you just check current job does it needs to execute when it perform ?!

Set up Redis maxmemory

Set maxmemory of redis.conf.
Otherwise, Redis will use server memory as much as it cans. It might cause to the server crash or some wizardly problems.

Using an initializer to separate rdb

Redis default number of databases is 16, 0 ~ 15.By default, Sidekiq uses 0.

Jobs will conflict as the two or more projects using Sidekiq at the same DB.

Although they may not cause exception or bug, it could get slowly if unrelated job too much.

To prevent you forget to separate DB in production just in case, you could setup DB URL after install Sidekiq immediately.

It is important to note that to configure the location of Redis, you must define both the Sidekiq.configure_server and Sidekiq.configure_client blocks. Put the following code into config/initializers/sidekiq.rb.

https://gist.github.com/lustan3216/e94e161ad43bb93e0c8e41ce2cdecfaa

REFERENCE

Do not use namespace in redis to separate rdb

Do not use namespace, not only using Sidekiq but also using Redis individual.

Client-side sometimes prefix key with namespace, e.g. “db1”, so the key “db1” don’t conflict with the key “db2”

The point is namespace increases the size of every key, and it will be expensive cost.

Here reference below is a definite proof from Mike Perham(Author of Sidekiq).

REFERENCE

Pass id for instance rather than instance

Pass instance id e.g. SomeWorker.perform_later(user_id) rather than SomeWorker.perform_later(user)

https://gist.github.com/lustan3216/4aa2df10a3840519cc32c7888b1152ff

What the pattern benefit is that can recheck instance exists or not as it performs.

Supposing we execute SomeWorker.set(wait_until: Time.current + 3.days).perform_later(user) and the user had already deleted possible in this case.

Then the job will be set in retry zone which makes the pointless loop.

REFERENCE

Stringify params

Do not put instance directly to be params, why?

• Put instance id can reduce the Redis DB size cost rather than an instance.
• Sidekiq client API uses JSON.dump to send the data to Redis.The Sidekiq server pulls that JSON data from Redis and uses JSON.load to convert the data back into Ruby types to pass to your perform method. Don’t pass symbols, named parameters or complex Ruby objects (like Date or Time!) as those will not survive the dump/load round-trip correctly.

Need a callback when jobs complete or success in a free way?

Batch is pro function in Sidekiq, so if you want a callback, you need to pay for pro version.

sidekiq-batch

Here is a free way gem to use.

https://gist.github.com/lustan3216/023d04c080f602f4eef0cf27e3deb30e

Set pool: 25 in database.yml

By default, one Sidekiq process creates 25 threads.

If you didn’t set pool: 25 in database.yml on production, it will cause lots bugs when large jobs.

The author of Sidekiq also recommends not set the concurrency higher than 50.

Prevent dead Lock

Due to Sidekiq do jobs concurrency, its has lots chance to happen dead lock.

Rails provide an easy to way to prevent.

  ad.with_lock do
    ad.send(action)
    ad.do_someting
  end

Set monintor

Mike Perham(Sidekiq builder) built this monitor because he didn’t like the existing tools that were available (e.g. monit, God and bluepill).

Inspeqtor

This one is Mike Perham recommendations:

• Use Upstart or Systemd to start/stop Sidekiq. This ensures if the Ruby VM crashes, the process will respawn immediately.
• Use Inspeqtor to monitor the CPU and memory usage and restart Sidekiq if necessary.

REFERENCE

Please give some applauses 👏👏👏👏 if this article makes you better with Sidekiq. Hint, applauses can leave to 50 😳.