Node.js Array processing: getting from 4 seconds to under 100ms
Nodejs is very fast when it comes to processing of arrays. If you have a simple array with 100.000 numbers it’s really fast. So everything was fine until I wanted to process a large Array of tree based data and add the data of the sub-elements to the main element. Something which happens quite often if you want to process permissions on a tree-based structure.
As I knew the performance of the pure array iteration in node.js i never expected that it could take more than 4 seconds to process my little tree (40.000 iterations overall). As i was not happy with the result i created a little cache mechanism for every element already being processed in the tree. This decreased the time to about 2 seconds. But that was still to long.
I tried using a lot of differen for implementations like _.for and for with a length cache and many more. This did not change a lot. I did not check for duplcates before pushing to the array cause i thought if i add a indexOf it will slow down the process. Actually it increased the performance and a _.inArray was also a bit faster. Afterwards the performance of our nested for construct was at about 700ms…… Then I realized that the size of the array has an major impact on the performance of a array push which I did very excessively in my code.
The solution was to create empty array instance before every for — loop and push to that instance. After the for — loop I concateinated the new instance with the “global” array. After I did that in every for — loop the needed time decreased to under 100ms.
Originally published at inspirationlabs.com.
