Scalable Background Tasks Worker
by Paulo Pereira
We are always looking at ways to improve our services for our customers — quality is paramount. Over the years, we’ve aimed to scale-up our infrastructure to cope with more demand, enabling us to become more responsive. However, we were not fully able to do that until we migrated to Azure Platform-as-a-Service (PaaS) and then worked on a mechanism that could handle heavy processes and be scalable — a background task framework.
Migrating to Azure PaaS meant direct HTTP connections could not take longer than 230 seconds. If they did, they would create a ‘500 Error message’. As all of the processes and calculations had to be completed on a web application, there was a subsequent impact on the users, making it more unresponsive as there was a lot of CPU power being used. We had been using the concept of a worker, which was another server that would help us to handle heavy workloads, but it was limited in its ability and cumbersome to use. For customers, this meant that if they had tried to push a big report through then their whole environment may have become slower — we wanted to rectify this.
As a technical product owner, I worked with my frameworks squad to build a background task framework. The frameworks squad belongs to the platform squads, and we provide frameworks for the feature squads to use and apply.
What we created was a framework where when the functionality is heavy processing, it will request a background action, it connects through the browser to the web application, and requests a background method execution from the background task orchestrator. This could be something like ‘I want a report’, so internally the orchestrator notifies the API that the user wants to execute something in this environment, and then we store that information in an Azure Storage Table. The request object with all of the execution information that enables the background processing is also serialized and stored in a blob container.
This then returns an ID to the orchestrator. The orchestrator then creates a record in Hangfire, a background processing tool, which is where the pool of workers are. Every request is handled by the workers using a load balance mechanism, so the workers then fetch the task to work on and notify the orchestrator that they are working on it. They update the orchestrator on the status of the task which feeds into the browser pooling process. At the end of the processing, the result is also serialized and stored in a container.
As every request using this framework is queued, we no longer have any 230-second issues, and we’re able to more effectively use infrastructure resources. We can also implement task expiration time, meaning that after a set period of time, the task expires and is deleted from the system. However, if within that time the user makes the exact same request, then we use the same output that was generated previously. So if a customer tries to constantly refresh, then it will only load the initial output. This ensures that our system remains efficient at all times.
The benefits of the project
This decision to go with Azure means that not only are we more responsive, we’re also able to scale up as and when we need to. Crucially, we are future-proofed as we do not need to keep adding servers or VMs to scale-up. Microsoft Azure makes the framework scalable, meaning that if we are in need of more we can quickly add more workers to the pool or remove them if they’re no longer being used. Importantly, as all of the heavy processing is carried out outside of the web application servers, Nmbrs will feel more responsive to our customers. There’s also more CPU power for the normal usage of Nmbrs.
In addition, this change makes it easy for developers to use the framework, whenever there is a new project or microservice required, they can use the framework to carry out tasks in the background and running on specific workers, even if they’re different machine sizes with more power they can handle these tasks better than they would have been able to previously.
The feedback from developers has been great; they say it’s easy to use and more secure. Right now, all of the heavy processes run on this framework, from API calls to reports, runs, you name it.
Since this project, we’ve already worked on our next task which is to deliver notifications. This means whenever a task is finished you’ll receive a pop-up notification. As always, we’re listening to our employees and customer feedback, and making sure that we provide the best service possible!
Would you like to know what if feels like building software used by more than 800K people monthly? Join us as a Senior .NET developer!
