Async loops, and why they fail! Part 1
Higher order functions such as
In this article we’ll study and produce alternative correct implementations for
reduce(), which are quite similar in some aspects. In the second part of the article we’ll deal with the complementary
reduce() functions; in the third we’ll work with
every(), and in the fourth and last article we’ll consider
By the way, we will be using the latest version of Node.js, including
.mjs modules, allowing us to use
Viewing the problem
In order to see what the problem is, let’s start by having a fake async call, that will just wait a short time (
timeToWait) and return a given value (
dataToReturn). For testing purposes, sometimes we’ll also want to be able to make the call fail, so we’ll include a third parameter (
fail) that will be false by default. We will use the code below for most examples.
We will also want a logging function including the current time, and we can make do with the following.
Now that we have these functions, let’s see some code. The following sequence works perfectly well — but we expected that, since there are no loops anywhere!
Running this, we get the results below, which are OK. The async calls don’t go out in parallel; the first requires 1 second, the next 2 seconds after that, and so on, so the whole experiment takes around 19 seconds.
13:05:28.264 START #1 -- sequential calls
13:05:29.269 data #1
13:05:31.270 data #2
13:05:34.274 data #3
13:05:39.274 data #5
13:05:47.283 data #8
13:05:47.283 END #1
We can also try this out with a common
for() loop, and it will also work — which is to be expected, since no higher order functions are involved either.
Results are similar; calls go out in sequence as before, and so on.
13:05:47.284 START #2 -- using a common for(...)
13:05:48.285 data #1
13:05:50.286 data #2
13:05:53.290 data #3
13:05:58.292 data #5
13:06:06.296 data #8
13:06:06.297 END #2
However, let’s now try with
Oops!! The loop ends before any async calls are done!
13:06:06.297 START #3 -- using forEach(...)
13:06:06.298 END #3
13:06:07.299 data #1
13:06:08.298 data #2
13:06:09.298 data #3
13:06:11.298 data #5
13:06:14.298 data #8
The unexpected problem is well known; for example, in MDN we read “forEach expects a synchronous function - forEach does not wait for promises. Kindly make sure you are aware of the implications while using promises (or async functions) as forEach callback.”
This kind of problem also affects
reduce(), and others, so let’s see how to work around this!
How can we solve the
forEach() problem? As we’ll be dealing with promises, the result of that method will be a promise itself. We want to go sequentially through the array, calling the provided callback each time — but not until the previous callback has finished. A simple way to manage this is to chain the new call to the previous one. We can use
finally() so we’ll be able to deal with failures (ignore them) as well.
We make do by using
.reduce() and starting with a resolved promise. For each element in the array, we call the async function in the
.finally() call for the previous promise. (We could also work with both
.catch() but we’d have to duplicate code.) After a promise succeeds, the next function call will go out, traversing the whole array.
In all cases, we’ll be giving two implementations for each function — one by adding to the Array.prototype (though modifying a prototype is not usually recommended…) and one as a stand-alone function, and you can select the one that you prefer.
Let’s see this alternative implementation work! We’ll have a
getForEachData() call that will get values from our mock API call. Just for variety, we’ll have the call fail if we pass 2 as its argument. Full code is below.
Both implementations produce the same kind of result, so let’s see just one run.
17:26:16.476 START -- using .forEachAsync(...) method
17:26:16.480 Calling - v=1 i=0 a=[1,2,3,5,8]
17:26:17.482 Success - data #1
17:26:17.482 Calling - v=2 i=1 a=[1,2,3,5,8]
17:26:19.484 Failure - error
17:26:19.484 Calling - v=3 i=2 a=[1,2,3,5,8]
17:26:22.488 Success - data #3
17:26:22.488 Calling - v=5 i=3 a=[1,2,3,5,8]
17:26:27.494 Success - data #5
17:26:27.494 Calling - v=8 i=4 a=[1,2,3,5,8]
17:26:35.503 Success - data #8
Success! The sequence of logs is what we expected: an initial START, then all five calls, and a final END. And, as a plus, a very similar algorithm will work as an alternative for
.reduce() — let’s see how.
Reducing an array to a single value using
.reduce() also requires going through all its values sequentially. I’ll admit, however, that calling a remote endpoint to do the reducing isn’t a very likely situation, but let’s just accept that for completeness’ sake.
The exact type of code we wrote above will serve — but we have the reducing process with the initial value, and each promise has to pass the updated result to the next call. We can write the following, then.
If you compare the
reduceAsync() code with the previous code for
forEachAsync() two things appear:
- we provide a promise, resolved to the initial value for reducing, to
- we aren’t using
.finallybecause we want to pass a value to the next promise; if the previous call was successful, we pass the updated accumulator, and if the call failed, we ignore it, and pass the (unchanged) accumulator.
We can see this work; the code below uses our new implementation.
Our (fake) reducing call just sums the accumulator and the new value. When the passed value is 2, the call “fails” instead. The result of both loops is similar; let’s see just one.
17:37:35.646 START -- using .reduceAsync(...) method
17:37:35.650 Calling - v=1 i=0 a=[1,2,3,5,8]
17:37:36.652 Success - 1
17:37:36.653 Calling - v=2 i=1 a=[1,2,3,5,8]
17:37:38.655 Failure - error
17:37:38.655 Calling - v=3 i=2 a=[1,2,3,5,8]
17:37:41.658 Success - 4
17:37:41.658 Calling - v=5 i=3 a=[1,2,3,5,8]
17:37:46.663 Success - 9
17:37:46.663 Calling - v=8 i=4 a=[1,2,3,5,8]
17:37:54.671 Success - 17
17:37:54.671 END -- 17
All values — except 2, which was ignored because of the faked failure — were added, and the final result is 17; we’re done!
What about using .reduceRight() with async calls? In reduceAsync(), just change .reduce() to .reduceRight(), and you’ll have your reduceRightAsync().
In this first article, we’ve seen that some higher order functions fail when working with async calls or promises, and we developed alternative implementations for
forEach(). In the next article in the series we’ll provide alternatives for
filter(), which also won’t work correctly if used asynchronically.
Code for all articles in the series is available at my repository:
Contribute to fkereki/asyncLoopsArticle development by creating an account on GitHub.
Don’t miss the rest of the series:
Async loops — and why they fail! Part 2
Other problems with async calls and loops; handling filtering and reducing
Async loops, and why they fail! Part 3