NodeJS: page through a result set with Promises and Streams

APIs normally ask clients to request results in pages, i.e. instead of asking for 1,000 results at once, the client should send 10 requests for 100 results each.

How to read all results from such an API with NodeJS?

Simple querying

We’re going to use node-fetch, which brings the browser’s fetch API to NodeJS:

const fetch = require("node-fetch");
const url = require("url");

Next, we add a method that fetches the results:

const ITEMS_PER_REQUEST = 10;
const fetchApi = () => {
  const request = url.parse('http://www.my-api.com', true, false);
  requestUrl.query = {
    count: ITEMS_PER_REQUEST
  };

  fetch(request.format(requestUrl), {
    method: 'GET'
  }).then(response => {
    if (response.ok) {
      return response.json();
    }
  }).then(json => {
    json.entry.forEach(item => console.log(item.title));
  });
};

When the promise of the fetch requests resolves, the result is outputted to the console.

There are two things missing: 1.) return the results to the caller (instead of just outputting them to the console) and 2.) page through the result set.

Let’s start by handing the results back to the caller!

Stream results to the caller

Instead of just returning the final list as an array, we’ll stream the results back whenever they are available. This is pure overhead right now with only one request. But as soon as we page through the result set, we’ll be able to return the items to the caller whenever a request returns. Streaming allows not only to returns results faster to the user but it also reduces the memory consumption because less results need to be buffered.

So, let’s do that!

First, we’re adding through2, a tiny wrapper around Node streams:

const through = require("through2");

Now, we can expand our fetchAPI method:

const ITEMS_PER_REQUEST = 10;
const fetchApi = () => {
  const stream = through.obj();

  const request = url.parse('http://www.my-api.com', true, false);
  requestUrl.query = {
    count: ITEMS_PER_REQUEST
  };

  fetch(request.format(requestUrl), {
    method: 'GET'
  }).then(response => {
    if (response.ok) {
      return response.json();
    }
  }).then(json => {
    json.entry.forEach(item => stream.push(item));
 
    // Signal end of stream
    stream.push(null);
  });

  return stream;
};

The function can be used by a caller to e.g. output the results to the console:

fetchApi().pipe(
  through.obj(function(item, enc, cb) {
    console.log(item);
    cb();
  })
);

Page through the result set

Finally, we need to page through the result set. There are two points to consider:

1. The API will likely not allow us to send 100s of requests in parallel. For simplicity, we therefore just execute our requests sequentially.
2. We don’t know how many results there are before receiving the response for our first request. Take for example an API following the Open Search specification, which would expose the total results as:

{
  totalResults: 624,
  itemsPerPage: 10,
  startIndex: 0,

  entry: [
    { title: "1st result" },
    { title: "2nd result" },
    { title: "3rd result" },
    ...
  ]
}

Sounds complicated? Leveraging the power of streams, this is actually surprisingly simple to implement:

const ITEMS_PER_REQUEST = 10;
const fetchApi = (startIndex = 0) => {
  const stream = through.obj();

  const request = url.parse('http://www.my-api.com', true, false);
  requestUrl.query = {
    startIndex,
    count: ITEMS_PER_REQUEST
  };

  fetch(request.format(requestUrl), {
    method: 'GET'
  }).then(response => {
    if (response.ok) {
      return response.json();
    }
  }).then(json => {
    json.entry.forEach(item => stream.push(item));
 
    if (startIndex + ITEMS_PER_REQUEST < json.totalResults) {
      // Stream next following batches
      fetchApi(startIndex + ITEMS_PER_REQUEST).pipe(stream);
    } else {
      // Signal end of stream
      stream.push(null);
    }
  });

  return stream;
};

What’s happening here? After we streamed out all results of the current request, we check if there are more results. If so, we recursively call the same function for the next batch (startIndex + ITEMS_PER_REQUEST) and pipe the results stream through to the caller. If not, we signal that the the end of stream was reached.

Happy coding!

Want to learn more about coding? Have a look to our other articles.