Async loops, and why they fail! Part 1
Mixing loops with async calls in JavaScript produces unexpected results; how to handle looping and reducing.
Higher order functions such as forEach(), reduce(), map(), and filter() are commonly used in JavaScript programming, but if you mix them up with async calls or promises, results probably won’t be what you would expect. The problem occurs, as we’ll see, only with these functions; “common” code won’t be affected.
In this article we’ll study and produce alternative correct implementations for forEach() and reduce(), which are quite similar in some aspects. In the second part of the article we’ll deal with the complementary map() and reduce() functions; in the third we’ll work with some() and every(), and in the fourth and last article we’ll consider find() and findIndex().
By the way, we will be using the latest version of Node.js, including Promise.allSettled(), Promise.finally(), and .mjs modules, allowing us to use import and export, so we’ll also get a chance to try out these newer features! Thus, even if you don’t get to actually require mixing loops and async calls, all this work will prove an interesting exercise in JavaScript coding.
Viewing the problem
In order to see what the problem is, let’s start by having a fake async call, that will just wait a short time (timeToWait) and return a given value (dataToReturn). For testing purposes, sometimes we’ll also want to be able to make the call fail, so we’ll include a third parameter (fail) that will be false by default. We will use the code below for most examples.
We will also want a logging function including the current time, and we can make do with the following.
Now that we have these functions, let’s see some code. The following sequence works perfectly well — but we expected that, since there are no loops anywhere!
Running this, we get the results below, which are OK. The async calls don’t go out in parallel; the first requires 1 second, the next 2 seconds after that, and so on, so the whole experiment takes around 19 seconds.
13:05:28.264 START #1 -- sequential calls
13:05:29.269 data #1
13:05:31.270 data #2
13:05:34.274 data #3
13:05:39.274 data #5
13:05:47.283 data #8
13:05:47.283 END #1We can also try this out with a common for() loop, and it will also work — which is to be expected, since no higher order functions are involved either.
Results are similar; calls go out in sequence as before, and so on.
13:05:47.284 START #2 -- using a common for(...)
13:05:48.285 data #1
13:05:50.286 data #2
13:05:53.290 data #3
13:05:58.292 data #5
13:06:06.296 data #8
13:06:06.297 END #2However, let’s now try with forEach()!
Oops!! The loop ends before any async calls are done!
13:06:06.297 START #3 -- using forEach(...)
13:06:06.298 END #3
13:06:07.299 data #1
13:06:08.298 data #2
13:06:09.298 data #3
13:06:11.298 data #5
13:06:14.298 data #8The unexpected problem is well known; for example, in MDN we read “forEach expects a synchronous function - forEach does not wait for promises. Kindly make sure you are aware of the implications while using promises (or async functions) as forEach callback.”
This kind of problem also affects map(), reduce(), and others, so let’s see how to work around this!
Looping
How can we solve the forEach() problem? As we’ll be dealing with promises, the result of that method will be a promise itself. We want to go sequentially through the array, calling the provided callback each time — but not until the previous callback has finished. A simple way to manage this is to chain the new call to the previous one. We can use finally() so we’ll be able to deal with failures (ignore them) as well.
We make do by using .reduce() and starting with a resolved promise. For each element in the array, we call the async function in the .finally() call for the previous promise. (We could also work with both .then() and .catch() but we’d have to duplicate code.) After a promise succeeds, the next function call will go out, traversing the whole array.
In all cases, we’ll be giving two implementations for each function — one by adding to the Array.prototype (though modifying a prototype is not usually recommended…) and one as a stand-alone function, and you can select the one that you prefer.
Let’s see this alternative implementation work! We’ll have a getForEachData() call that will get values from our mock API call. Just for variety, we’ll have the call fail if we pass 2 as its argument. Full code is below.
Both implementations produce the same kind of result, so let’s see just one run.
17:26:16.476 START -- using .forEachAsync(...) method
17:26:16.480 Calling - v=1 i=0 a=[1,2,3,5,8]
17:26:17.482 Success - data #1
17:26:17.482 Calling - v=2 i=1 a=[1,2,3,5,8]
17:26:19.484 Failure - error
17:26:19.484 Calling - v=3 i=2 a=[1,2,3,5,8]
17:26:22.488 Success - data #3
17:26:22.488 Calling - v=5 i=3 a=[1,2,3,5,8]
17:26:27.494 Success - data #5
17:26:27.494 Calling - v=8 i=4 a=[1,2,3,5,8]
17:26:35.503 Success - data #8
17:26:35.503 ENDSuccess! The sequence of logs is what we expected: an initial START, then all five calls, and a final END. And, as a plus, a very similar algorithm will work as an alternative for .reduce() — let’s see how.
Reducing
Reducing an array to a single value using .reduce() also requires going through all its values sequentially. I’ll admit, however, that calling a remote endpoint to do the reducing isn’t a very likely situation, but let’s just accept that for completeness’ sake.
The exact type of code we wrote above will serve — but we have the reducing process with the initial value, and each promise has to pass the updated result to the next call. We can write the following, then.
If you compare the reduceAsync() code with the previous code for forEachAsync() two things appear:
- we provide a promise, resolved to the initial value for reducing, to
reduce() - we aren’t using
.finallybecause we want to pass a value to the next promise; if the previous call was successful, we pass the updated accumulator, and if the call failed, we ignore it, and pass the (unchanged) accumulator.
We can see this work; the code below uses our new implementation.
Our (fake) reducing call just sums the accumulator and the new value. When the passed value is 2, the call “fails” instead. The result of both loops is similar; let’s see just one.
17:37:35.646 START -- using .reduceAsync(...) method
17:37:35.650 Calling - v=1 i=0 a=[1,2,3,5,8]
17:37:36.652 Success - 1
17:37:36.653 Calling - v=2 i=1 a=[1,2,3,5,8]
17:37:38.655 Failure - error
17:37:38.655 Calling - v=3 i=2 a=[1,2,3,5,8]
17:37:41.658 Success - 4
17:37:41.658 Calling - v=5 i=3 a=[1,2,3,5,8]
17:37:46.663 Success - 9
17:37:46.663 Calling - v=8 i=4 a=[1,2,3,5,8]
17:37:54.671 Success - 17
17:37:54.671 END -- 17All values — except 2, which was ignored because of the faked failure — were added, and the final result is 17; we’re done!
What about using .reduceRight() with async calls? In reduceAsync(), just change .reduce() to .reduceRight(), and you’ll have your reduceRightAsync().
Summary
In this first article, we’ve seen that some higher order functions fail when working with async calls or promises, and we developed alternative implementations for reduce() and forEach(). In the next article in the series we’ll provide alternatives for map() and filter(), which also won’t work correctly if used asynchronically.
References
This article is partially based on Chapter 6, “Programming Declaratively — A Better Style” of my “Mastering JavaScript Functional Programming” book, for Packt; some implementations are different.
Check MDN for the description of array.forEach(), array.reduce(). and array.reduceRight().
Code for all articles in the series is available at my repository:
Don’t miss the rest of the series:
