Why optimize ? The Power of Batch Operations, Batch Processing, and Concurrency

In today’s interconnected digital world, the efficient management of bulk operations, batch processing, and concurrency is crucial for optimizing performance and resource utilization. These concepts play a pivotal role especiallu in software development. In this article, I will share my personal experiences with bulk operations, batch processing, and concurrency, providing specific examples, challenges faced, and lessons learned as a Backend Engineer.

My Journey with Batch Operations

My recent experience with batch processing was from developing a Mass Broadcast Message feature. As a Backend Engineer, the term “Batch”, “Bulk”, and “Concurrency” is very common especially when handling big operations. The past week, I have been give the opportunity to develop a Blast Message feature for our messaging application “SalamChat”.

Challenges Faced

At first it seems quite simple, just apply concurrency when looping sending & writing those messages to Database. But the problem arose when you realize that looping the create operation to the database is limited and that doing the operation one by one with a pause to not overload the database connection would take me 2 hours to complete.

The Bulking helped med to face the looping of creating messages instead of doing it one by one I managed to insert them in a big query of creating operation.

Meanwhile Batching helped me to not bulk all messages at once (which would also take a lot of resources in an instance). By batching them into smaller messages per iteration, I’d only have around 20 iteration of small sized bulk operations that would not freeze the server operations.

Finally running the batch concurrently so they all run asynchronously would speed up the task making the ETA around 5 minutes to finish processing all messages (from 2 hours of process).

Lessons Learned

1. Efficiency through Batching : Batching tasks can significantly enhance performance by dividing those big tasks into smaller size operation and running them concurrently, the smaller size gives the server more room to handle other API endpoints while doing the smaller batches.
2. Time saving using concurrency : Integrating concurrency into the batch operations made it so that they can run in parallel, and since the operation is quite lightweight it would still be lower resource usage compared to one big chunk of hundred thousands insert operation.
3. Bulking to reduce stress load : A single insert operation might seem fast if it’s just one or a hundred single insert operations. But when we talk about mass data, we need a way to compress this big load. Transaction might help but to loop each transaction one by one would still take some resources from the machine. Best way to reduce the load is by combining transaction to the database with bulking each operation. In my case i was able to reduce thousands of loop insert operation to 20 Batch of loop insert operation with each loop containing a smaller ammount of insert operation in one query.

Future add-ons

1. Scaling the batch dynamically : Dividing the batches dynamically is a nice feature as well, since the userbase might grow, it’s needed for the codebase to adapt to the demands.
2. Putting min — max bulk into remote config : By applying remote config that can be easily altered or modified when needed is really useful as the applications demands grow.

Conclusion

In conclusion, my personal journey with my last feature development was a nice trip to seek efficiency and optimization (every programmer’s friend). Reminding us why we are developers, finding the easiest solution is alright, but a proper optimized one should never be forgotten. This article is just one of the many ways we can implement an optimized driven solution.

Although, searching for the most optimized and resource efficient way doesn’t always have to be our first choice. Easy solutions are fast to achieve, but I just want to point out that it has to be temporary (and searching for an optimized code makes me feel alive as a coder).

By sharing my experiences and insights, I hope it inspire others to not forget an important practice for us coders, enabling them to push creativity to complexities of challenges or tasks ahead with confidence and proficiency.

This Medium article is a reflection of my personal experiences and a comprehensive exploration of the benefits, challenges, and best practices associated with batch operations, batch processing, and concurrency. It aims to equip readers with valuable insights for effectively managing and implementing these concepts in their own way. Also big thanks to my Salam Team for supporting me and each other to accomplish these hard tasks together.