Managing concurrent HTTP requests in node.js using .map(), .reduce(), and Promise.all()

I recently ran into a programming challenge involving concurrency with some interesting learning outcomes on asynchronous functional programming in node.js.

The main challenge centered around submitting a set of queries to a json API, checking the total length of the paginated query response, requesting the rest of the query response pages, and parsing those responses into a single merged output, all while minimizing time spent waiting for synchronous HTTP responses.

The specific problem at hand was to find out how many football matches ended in a draw in a given year. To to this, the API was queried with then number of goals a game could be tied on, from 0 to 10. Some (but not all) of those responses were paginated, so it was necessary to make multiple requests to get the full results.

The number of games ending in a draw, for a given page of results, was found by checking the response data array length. In order to check if further requests are required, the first page of responses also returns the total number of paginated query results.

While it would be possible to use a set of ‘for loop to make each request synchronously, this would end up being quite slow. Instead…

In order to get the total number of games tied by a given score:

• The function makes a blocking await request for the games count from the first page of results, and the number of pages in the paginated results.
• The total pages response integer is used to create an array of integers, with the first page filtered (to avoid redundant network requests).
  e.x. ( 4-> [0,1,2,3] -> [1, 2, 3])
• This integer array is mapped to an array of promises (all async functions have a return type of promise),
• This promise array is concatenated with the promise for the count from the initial query.
• The code block uses Promise.all() to resolve this array of promises into an array of integers
• And finally it returns the summation of this array using reduce().

This gives the number of games tied by a single score in a given year (ex. The number of games in 2011 where team1goals=team2goals=1), but the complete result needs games tied on all scores from 0 to 10. To do this:

• All scores that games could be tied on are mapped into an array [0…10].
• This array of integers is mapped to an array of promises.
• The code block waits for Promise.all() to resolve this array of promises to an array of integers.
• On resolve, the resulting array of integers is summed using reduce() to return a total.

Finally, it all gets put together in the snippet below. Let me know if you see any issues or improvements to be made!

An update after time spent as a professional developer: Heavy use of reduce() is not a very good idea in a professional setting. Use of nested reduce functions as shown here is even less so, but it’s still pretty fun to solve things like this. I would never again attempt this kind of solution without strict typing (typescript or something similar).