C#: ActionBlock to process a large collection in small sets
Last week I got stuck in a task where I have to generate a report for half a million users. And for each user, there is n number of associates. When I tried to process more than 500 users together, it eats up some system resources. Another 3rd party API in the downline is not able to handle the load. For each user, I have to fetch data from SQL Server, MongoDB, Elastic search and Orient Graph database, and finally combine them together.
Sometimes we have some assemblies or 3rd party DLLs, we cannot get rid off and we have to code accordingly. I remember, once I used an assembly that has a GET method to query the database by passing parameters, but MS SQL does not support more than 2100 parameters. So, I have to call that method multiple times, passing 2100 parameters at a time.
Design
One way to do it is by using the ActionBlock and LINQ.
Please refer following code, it is not clean, just for the reference.
There is a for loop, that takes out 100 users at a time from the huge collection and add that small subset of users to ActionBlock for processing.
I set the degree of parallelism to 5, so that we can execute 5 threads in parallel, making the total count of 500.
In code, I added the results of each request to thread safe ConcurrentBag.
Conclusion
It is a small and very simple way of doing this.
Please check for yourself, if we can use BufferBlock instead of ActionBlock or not. Why or Why not.
Cheers.
More Readings
