Performance Modeling in Mendix — Batch processing
--
Even though some people might say that low-code platforms like Mendix restrict the developer in what can be built, it is very easy to build applications supporting complex business processes. But just as with any other programming language you need to keep a keen eye on performance.
In this series of blogs, I talk about several aspects that play a big part in keeping your Mendix application performing the way it should.
Part 2: Batch processing
Batch processing can mitigate a variety of potential performance issues. Whether it is data cleanup, conversions, generating reports, or just having to update a lot of objects in one of your business processes; working with batches will increase the performance of your application.
Batch processing is something you don’t want to bother the end-user with and is therefore heavily linked with the decoupling or separating of processes in multiple steps. As most batch processes are run via scheduled events, you will see in the following examples that almost all of them have an inherent separation of processes implemented.
In this blog, I will focus on the batch processing part, because that is where the performance benefits come from. I will explain in which situations batch processing increases performance and I will go into some more detail regarding use cases and types of implementations for batch processing. While discussing the different scenarios I will also add some tips, tricks, and things to look out for when implementing your batch processes.
Types of batch processes
Batch processes can be implemented for a number of reasons. It could be that you don’t want to bother end-users with having to wait for a process to finish. So instead of executing it in the same transaction, you set a processed attribute on an object to a certain status, and implement a scheduled event that periodically checks whether there are any objects that need processing.
It might also be that you want to spread the processing of incoming messages over a greater period of time, in order to not overload the system. Incoming messages are saved and again, periodically picked up by a batch process.
A third option is conversions, which most developers are all too familiar with. Often when new functionality is added, old data needs to be updated to match the new way of working. Either by setting a default value or for example, recalculating every order in the system to set a newly added attribute.
Single microflow execution for batch processes
Most of the Mendix consultants will remember being taught the following structure as a default way of building a batch process in Mendix:
There might be some variations where you clear the list before you repeat the process, increment the offset for the next retrieve, and/or create an extra commit list to only commit once when the entire process is finished.
These types of microflows need to be executed perhaps only once a day because the entire list of objects is processed in one go.
If we’re completely honest, however, this is not really a batch process. Because a single microflow is executed in a single transaction, the process will become slower for each subsequent retrieve that is performed in this microflow.
You also can’t guarantee how long the process will take, because it depends on the total amount of objects that need to be processed.
Personally, I’m not a fan of these kinds of batch processes, but I will give you a tip that might save you some unexpected strange behavior:
When implementing a batch process like the one above, it is very important to check whether the process that you execute for a retrieved object, has an influence on the attributes or associations on which the object is retrieved.
This is important in order to determine whether you need to use an offset or not.
If the processing of an object influences whether it will match the XPath constraints of the next retrieve, you don’t need to use an offset, because the object is not retrieved, and therefore does not need to be skipped.
For example:
Batch deletes don’t need an offset if the delete action is performed before the next retrieve. If you are working with a processed attribute, you don’t need an offset if the commit action is performed before the next retrieve.
Rapid microflow execution for batch processes
To make sure that each execution of a batch process takes the same and predictable amount of time, you could decrease the interval between executions and only perform one retrieve per execution.
With this implementation, you are actually processing the objects in batches, because each retrieve, process, and commit sequence is executed in a separate transaction.
Because this implementation means that you need to execute the process multiple times, often in quick succession, it might also mean that you need to implement custom functionality to make sure the execution can only take place on those moments of the day that it won’t influence other processes.
Create a batch-process entity on which you can set the allowed range of hours of a day when the process can run.
With this setup you can for instance ensure that a batch process runs every 5 minutes, but only between 1:00 and 4:00 in the night for example. Of course, the scheduled event will be executed throughout the day, but only a simple check will be executed to determine that the actual process should not be executed.
Summary & What to do next
Processing in batches can spread the load on an application, it can remove waiting time for end users and if implemented correctly, will greatly decrease the number of database transactions that are executed.
In part 1 of this series, I talked about indexes to improve the speed of retrieves. Keeping your data clean, with a batch process, will definitely also help with improving the speed of your application.
Go take a look at any existing batch processes in your application that might benefit from the rapid microflow execution way of implementing. Also, take a good look at processes that might not need to be executed directly when the end-user puts in the request (and can therefore be scheduled for batch processing).
Finally, take a look at when and how the data in your application is being mass processed or cleaned up and if these processes are executed enough times, should be split up, or executed at other moments during the day.
Read More
From the Publisher -
If you enjoyed this article you can find more like it on our Medium page. For great videos and live sessions, you can go to MxLive or our community Youtube page.
For the makers looking to get started, you can sign up for a free account, and get instant access to learning with our Academy.
Interested in getting more involved with our community? You can join us in our Slack community channel or for those who want to be more involved, look into joining one of our Meetups.
