Workers patterns with hangfire
In the past year, at my work, we have used hangfire in production for a number of projects, however only recently we updated a to 1.5.7.
The 1.5.0 is a big release. I feel i am really late to the party, but (Oh, Boy!), q⁰^p, So many fruity features. And below are a few things which made me really excited and i think worth a bit more mentioning.
Taking control of background processes
Most significant update of hangfire since 1.5 is the NEW IBackgroundProcess and BackgroundProcessingServer. (at least it is relatively “new” to me)
It enables developers to de-compsite the BackgroundJobServer into processes, and creating custom process. (http://hangfire.io/blog/2015/10/01/hangfire-1.5.0.html. Currently the documentation is a bit lacking for this important feature on its offical doc)
Essentially Hangfire BackgroundJobServer (which is readily available since release 1.0 and still in valid today) now is in composition of following 3 BackgroundProcesses*:
- Worker (Executing jobs)
- DelayedJobScheduler (enqueue jobs from scheduled state into target queue)
- RecurringJobScheduler (create and enqueue jobs into target queue)
If you think above workflow is lacking the feature you need, you can even create your own by implementing IBackgroundProcess then add it in to your BackgroundProcessingServer
* Note: There are other hangfire process implementing IBackgroundProcess, however they are more generic service processors. You can checkout their source in github (https://github.com/HangfireIO/Hangfire)
What does this means?
Instead of initialize BackgroundJobServer instance with all 3 background processes on each server node. We now can separating roles and responsibilities between boxes. (Commanders and Workers) This give us a greater flexibility in architecturing our services, instead of making tech decision as a whole, now we can shift our focus on the nature job (problems) itself and choose the best fitting tech stacks that suits.
On the “Commander side”, its main responsibility is managing job states and queues. It doesn’t need any business logic implementation, all it need is the Job (“Worker”) interface contract.
- queue jobs into corresponding queue on request. (via api, fire and forget)
- queue recurring or scheduled job when time condition is met.
- Switching job state status when on fail and retries
Hence jobs state management and enqueuing is done by Commander, workers can be distributed to multiple processors (server nodes), All workers nodes need to do is watching assigned queue, and carrying out the real job processing. This will be where concrete implementation of jobs sits and being executed.
As a result, if we want to change logic of processing certain job. No change will be required on the Commander side. Based on the job priority and load we can even assign certain types of jobs in one server, and some other jobs on the other. i.e. when certain jobs are on high demand, we could deploy it in a separate and only scaling up the target worker nodes.
The other big plus is, based on the background process added to the server, we now have the ability of fine tune our queues by allocating different number of worker processor watching it. one useful use case is: when there is a api throttling based on concurrent connection numbers, by setting a specific worker number on the designated queue can make our life so much easier.
The future, containers
The .Net core and the docker contains is making me really excited about the direction of .net tech stacks. Unfortunately, hangfire.core is not supporting .net core yet, however the development is under its way. Imaging we make each worker processor into docker images and scaling up and down? (It is giving me goosebumps, just by thinking about it)
