Monitoring and Reporting Errors from Sidekiq Jobs
Sidekiq is famous (in the Ruby world) as an easy and simple background job processing solution. Sidekiq covers almost all the use cases that you expect from a background job processing solution but sometimes you need something that is not supported out of the box. That’s when Sidekiq middleware comes in the picture, middleware can extend Sidekiq to run your code before and during job life cycle.
We, at Clarisights, use Sidekiq extensively for background jobs. Just like any other piece of software, sometimes background jobs fail. When that happens we want to know why and how that happened.
Why do we need this?
We wanted to capture every unhandled error from background jobs. We collect all errors in our central error reporting tool (Sentry). Our application is configured to use sentry but we were only getting errors when all retries for a job were exhausted. It was happening because Sidekiq captures any exception occurred in a job (that’s how it decides that a job failed) and uses it to retry failed jobs. It doesn’t raise after capture unless retries are exhausted which will happen after a lot of retries (you can configure retry policy).
Sidekiq Sentry Middleware
Sidekiq has notion of server-side and client-side middleware, Server-side middleware runs around job life cycle while client-side middleware runs before pushing a job to Redis.
We wrote this server-side middleware to report unhandled errors from jobs. If a job fails, our middleware will capture an unhandled exception and report that to Sentry (can be easily adapted to report to other sources as well). After reporting, we will raise that exception so Sidekiq & other middlewares down the chain know that the job failed.
Raising captured exception again is important here because, if you don’t raise, Sidekiq or any other middleware down the chain will think that job completed successfully.
How to use it
Add it in Sidekiq middleware folder and register it in Sidekiq configuration.
It should work out of the box, provided you have Sentry (or any other source where you want to report) configured in your application.
Final Thoughts
After we started using Sidekiq error reporting, we were able to minimize the impact of failing background jobs. Now we have more visibility into our background jobs. When jobs fail we get alerts with the stacktrace, which helps a lot during debugging. Early detection results in an early fix, which in turn means minimal impact on users.
