Infinite Scrolling in React using Apollo and React-Virtualized — GraphQL Cursor Pagination

What is infinite scroll?

Before we jump into the technical details, let us first understand what Infinite Scroll is.

Have you ever noticed when you browse on website and as you keep scrolling down, more content is loaded dynamically to the page?

Let us take the best example of Facebook. As you keep scrolling down on the news feeds on your timeline, have you noticed how more contents keep getting loaded dynamically?

This is basically what Infinite Scrolling (Endless Scrolling) is.

It is an alternate way from the earlier pagination technique. And is quite popular these days.


Why am I writing this article?

I was looking at some good examples to help me get started with setting up a GraphQL server which will handle a GraphQL cursor based pagination and to consume it in a small React, Apollo-Client web app.

Unfortunately I found very few to no articles/examples which shows how to setup cursor based pagination end to end, i.e the server side setup as well as the client.Also most of the articles were around offset based pagination.

The GitHunt example provided by Apollo is really good, but I had two issues with it

  1. It is a complex example (a really good one, but complex), and if anyone is trying to get started with quickly, this may not be what they would be looking for.
  2. It covered off-set based pagination. (Not a big deal because Apollo Documentation shows how to do cursor based navigation clearly anyways).

Why I prefer Cursor based pagination?

Well for most use cases, there really is no difference between using a cursor-based or offset-based pagination.

But cursor based pagination is recommend in both the GraphQL as well as Apollo documentation.

Excerpt from Apollo Documentation:One downside of pagination with numbered pages or offsets is that an item can be skipped or returned twice when items are inserted into or removed from the list at the same time. That can be avoided with cursor-based pagination. “

So what will this article cover?

A very basic app I have created:

There is nothing much to the app, but just the logic around infinite scroll and cursor pagination :)

The stack I have used:

This article will cover the following:-

On the server:

  1. How to create a simple GraphQL server using Express and NodeJS and retrieve data from local MongoDB.
  2. How to create Schema for Cursor Pagination in GraphQL

On Client:

  1. How to retrieve data using pagination from GraphQL Server using Apollo Client
  2. How to implement infinite scrolling using React-Virtualized.

I will cover in detail about each in the subsequent sections. Also as you see, since I have logically broken down each section, it makes it easy for you to replace each section with some other technology or framework as you prefer.

Just use what you need out of this article 😃

Example, say you want to:

  • Replace GraphQL server created over NodeJS with a python based GraphQL server, the rest of the section would still be helpful to get the infinite scrolling working.
  • Or you want to replace Cursor pagination to Offset pagination, then skip only that section and rest of the section would still be helpful.
  • Similarly you would like to use Relay instead of Apollo or maybe some other library, or maybe use AngularJs instead of React, then only that section would be irrelevant for you in this article.

Understanding each of the sections.

How to create a simple GraphQL server using Express and NodeJS and retrieve data from local MongoDB

If you have never setup a GraphQL server on Node.Js, then refer this article I did earlier on Medium.

The Medium article covers:

1) Setting up MongoDB locally on your machine.

2) Setting up a basic GraphQL server using Express and Node.Js


How to create Schema for Cursor Pagination in GraphQL

Once you have setup a very basic GraphQL server as mentioned above.

Let us proceed into the next steps.

  • First we need to setup the Mongoose Schema/Model that we will use for the MongoDB

In the Medium article I mentioned above, TodoList collection is used, but for this example we use GraphqlPagination collection with the below structure:

{
"id" : <id_number>,
"item" : <item_text>
}

This is a very simple structure and the schema and model are as below:

/*./mongoose/GraphqlPagination.js*/

import mongoose from 'mongoose';
var Schema = mongoose.Schema;
// create a schema
var gpSchema = new Schema({
id: Number,
item: String
}, {collection:"GraphqlPagination"});

var graphPageModel = mongoose.model('graphPageModel', gpSchema);
export default graphPageModel

If you want to understand what Schema and Model in Mongoose are, click on the respective links to learn on it.

  • Next, we setup the actual GraphQL schema, which will be used to defined the GraphQL query we use for cursor pagination.

Great! so now we have setup the Mongoose Schema and Model and we know how the documents on MongoDB will look like.

  1. Setup the GraphQL Schema for our query that will do cursor pagination.

The schema was heavily inspired from this query used in GraphQL official documentation.

The cursor pagination query we will use is as shown below

mainQuery(first:$first,after:$after) {
totalCount
edges {
cursor
node {
id
item
}
}
pageInfo{
endCursor
hasNextPage
}
}
The schema for this is around 200 lines of code, so I will not post it here, but you can have a look at it in this link.

But of course, I am not going to leave you hanging there with just the query. I will break down the query and walk you through the resolve function of the GraphQL Schema.

2. Before I break down the query, let us first see what are the inputs I am passing.

mainQuery(first:$first,after:$after)

$first — This is basically setting the limit on the number of rows I want GraphQL to retrieve each time

$after — This is where we use Cursor to say getting me all the records after this cursor.

So putting them together, you can roughly translate the query as

Hey mainQuery retrieve me $first records $after this cursor

Example:- Hey mainQuery retrieve me 10 records after 100 ($after is the cursor which is id field from GraphqlPagination collection in our example).

And next time, hey mainQuery retrieve me 10 records after 110, and so on…

And this is the magic behind making a cursor pagination query in GraphQL :-)

Oh an a very important point to note — GraphQL team suggests to encode the cursor as base64

Alright, now you know what is the input we pass, and why we do it like this.

3 . Probably we should get on with the query and break it down and explore the resolve function.

  • Part 1 and 2 of the query will be explained together — Node & Edges

We query the GraphqlPagination collection using mongoose to retrieve the documents which are greater than the cursor we provided as input (after) and limit the value based on the input variable first.

We use mongoose cursor. And the cursor while retrieving each document, based on the cursor event data, we build the node(part 1 of the Query) and edge(part 2 of the Query).

Notice how we encode the cursor to base64.
  • Part 3 of the query — TotalCount

We run another query to get the total count, and resolve the promise.

  • Part 4 of the query — PageInfo

This is within the same promise block of part 1 & 2 of the query.

During the event data of the mongoose cursor, we build the node and edges.

And during the event end, i.e the end of the mongoose cursor, we build the PageInfo and resolve the promise.

  • Stitching all the parts together

If you noticed above, we have two promises, when both are resolved, we build the final output for mainQuery.

I have used promise since the queries run asynchronously in mongoose, and it was easier to handle the query result via promise and return the final output.


How to retrieve data using pagination from GraphQL Server using Apollo Client

The documentation provided by Apollo is really good, therefore I won’t go into the very fine details, but only highlight the query options that I am using to create cursor pagination.

The earlier section explained about the query schema we have created on the GraphQL server which we will be used for cursor pagination, with the $first and $after input variables.

But it is in this section we will cover how we will change the value of after so that we pass the new cursor value to the server, and retrieve the new set of result and repeat the process as and when required by passing new $after value and getting new result set from where we left last. This is key to Cursor Pagination

Though we discussed about the query already, while we covered the schema on server, this is how I will pass the query from the code using graphql-tag

/* src/js/containers/PaginationContainer.js*/
const PaginationContainerWithData = graphql(MyQuery,configObject)(PaginationContainer)
export default connect(mapStateToProps)(PaginationContainerWithData)

Notice above that I am passing configuration object(configObject) along with the query to graphql container?

This configuration object is what we are going to cover in detail in this section. This is an important link between the server and the client from the pagination perspective.

  1. The options props is where we pass the values for the variables. In our example, I have set first = 5 always, but after value will be the last cursor value from previous result set.
  2. props is the property which we can use to modify the props before we send to our child components (In our example it is PaginationContainer).

Notice one of the props returned in data called fetchMore? This is provided by the graphql HOC we used in the above code snippet.

And we use this fetchMore in the function called loadMoreRows, which will be used as callback when we need to load more rows.

There are two important properties for fetchMore:

  • variables: This is where we pass the new value of after
  • updateQuery: This function is used to merge your previous result with the current result.

3. Finally we return only those props that are required by the child components.

How to implement infinite scrolling using React-Virtualized.

I have used React-Virtualized’s InfiniteLoader and List, to create infinite scrolling.

Notice the loadMoreRows, used here? InfiniteLoader has a callback function loadMoreRows , which will callback the loadMoreRows function we created in the configuration object in the earlier section and passed it here as a prop.

So as and when we keep scrolling down, InfiniteLoader fires this callback method to get more rows.



Setting up

If you are interested to try this yourself, then please follow the steps to set this code in your repo easily.

  1. Clone the project into your local folder
git clone https://github.com/Gethyl/GraphQLPagination

2. Change to that folder.

cd ./GraphQLPagination

3. Install the dependencies by issuing yarn install or npm install


Running the example

Just issue the below commands and you are good to go 😉

Client

To start the WDS (Webpack Dev Server), run the below command.

yarn start

Server

Since this example uses MongoDB,

  • Check if MongoDB service is running by issuing the below command.
service mongod status
  • If the status of the service is Inactive, then bring up the service by issuing
sudo service mongod start
  • Now that the MongoDB service is up and running.
It is not required to create GraphqlPagination collection in MongoDB, because first time when you add the item and if the collection is not there, MongoDB will create the collection.
  • And finally start the server by running
yarn run server

Setting up Data on local MongoDB

Aah yes, the boring part!

Assuming you have cloned my github repo and have both the server and test running, and you want to setup the data, and if you are wondering if there is an easy way to get large number of documents loaded to your collection in a single go.

I feel you ❤️

Just hit the following url http://localhost:5000/insertdb and this will insert around 100 documents in GraphqlPagination collection. 😉


Conclusion

I had to read a lot from the GraphQL and Apollo documentations and understand how to set up a cursor based pagination and use it for Infinite scroll and I am glad I did as it was a very good learning for me.

I hope this article is of good use for you.

Until my next article…..

Happy Coding!! 🖒

Oh and if you think this article benefited you, then please hit ❤️️