Infinite Scrolling and Basic Cursor-based Pagination

Hi!

In this article, you will be learning about the basic idea of infinite scrolling and why it is not only a frontend concept.

As a fellow developer, for sure, you already have an experience in making, or at least cloning, a social media site just like Facebook. And, just like Facebook, one of its notable features is having a homepage where users can interactively post or just scroll without end. Nowadays in fact, it already is a common feature on social media sites to have an infinite scrolling user interface, so as a beginner like myself, you must be familiar with it.

Now, if you’re interested to learn more, let’s try to break down this feature to have a better grasp on how to implement it.

First, create a webpage that shows a list of posts where the user can scroll down to the bottom of the page;

Second, when the user reaches the bottom of the page, automatically trigger the API to fetch new posts and then the user can just continue scrolling down the page as the cycle continues; and

Third, create an API to return a list of posts based on the requested page parameter.

To better picture out this scenario, let’s show some examples.

First, let’s create a simple vue application that during the first page load, it will list posts in the webpage;

// App.vue
<template>
  <div 
    class="result" 
    v-for="comment in comments" 
   :key="comment.id">
    <div>{{ comment.id }}</div>
  </div>
</template>
<script setup>
import { ref, onMounted } from "vue";
let page = 1;
const comments = ref([]);
const load = async () => {
  const response = await fetch(
    `http://localhost:3001/comments/page/${page}`
  );
  const json = await response.json();
  comments.value.push(...json);
  page++
};
onMounted(async () => {
  await load()
})
</script>

Next, simply install v3-infinite-loading component to fetch the succeeding pages whenever the user reaches to the bottom of the page; and

// App.vue
<template>
  <div 
    class="result" 
    v-for="comment in comments" 
   :key="comment.id">
    <div>{{ comment.id }}</div>
  </div>
  <InfiniteLoading :comments="comments" @infinite="load" />
</template>
<script setup>
import { ref } from "vue";
import InfiniteLoading from "v3-infinite-loading";
import "v3-infinite-loading/lib/style.css";
let page = 1;
const comments = ref([]);
const load = async ($state) => {
  const response = await fetch(
    `http://localhost:3001/comments/page/${page}`
  );
  const json = await response.json();
  comments.value.push(...json);
  $state.loaded();
  page++;
};
</script>

Lastly, create a simple backend server using expressjs.

Also, write an API that returns a list of comments from the database based on the requested page.

// index.js
const express = require("express");
const app = express();
const bodyParser = require("body-parser");
const cors = require("cors");
const db = require("./queries");
const port = 3001;
app.use(cors());
app.use(bodyParser.json());
app.use(
  bodyParser.urlencoded({
    extended: true,
  })
);

app.get("/comments/page/:page", db.getPostsByPage);

app.listen(port);

// queries.js
const Pool = require("pg").Pool;
const pool = new Pool({...});
const getPostsByPage = (request, response) => {
  const page = parseInt(request.params.page) - 1;
  const limit = 100;
  const sql = `
    SELECT *
    FROM posts
    ORDER BY id ASC
    OFFSET ${limit * page}
    LIMIT ${limit}
  `;
  pool.query(sql, (error, results) => {
    if (error) throw error;
    response.status(200).json(results.rows);
  });
};
module.exports = {
  getPostsByPage
};

Now, with everything right into place, you will have a simple application that will give you an infinite scrolling user experience.

In a real world scenario, your database will continue to grow rapidly as more users write or remove comments. And with the current pagination implementation, you are bound to encounter issues.

In most cases, low offset queries are not slow, but, If you reach “LIMIT 100000, 20”, you’re actually requesting the database to go through 100020 rows and return only the last 20. This will cause the app to feel laggy as you scroll down page by page.

To deal with this, we need to implement a different pagination mechanism using cursors.

Cursor-based pagination is using a unique column (or columns) on a table as an index to a specific row. This cursor can then be used for the subsequent requests where the server returns results after the given pointer. Unlike offset, this method does not scan a large dataset only to use very few.

To Implement this, let’s modify our sample app.

• Select a unique column that we can use as cursor. In this example, let’s use the id field.
• Modify the API to accept the cursor parameter instead of page number which the client would get in the response from the previous request.

// index.js
...
app.get("/comments/next/:id", db.getPostsByCursor);
...

// queries.js
...
const getPostsByCursor = (request, response) => {
  const nextId = parseInt(request.params.id);
  const limit = 100;
  const sql = `
    SELECT *
    FROM posts
    WHERE id > ${nextId}
    ORDER BY id ASC
    LIMIT ${limit}
  `;
  pool.query(sql, (error, results) => {
    if (error) throw error;
    response.status(200).json(results.rows);
  });
};
module.exports = {
  ...,
  getPostsByCursor
};

Next, let’s update our vue application to store id of the last element from the API and use it as a cursor for the next request.

// App.vue
<template>
  <div 
    class="result" 
    v-for="comment in comments" 
   :key="comment.id">
    <div>{{ comment.id }}</div>
  </div>
  <InfiniteLoading :comments="comments" @infinite="load" />
</template>
<script setup>
import { ref } from "vue";
import InfiniteLoading from "v3-infinite-loading";
import "v3-infinite-loading/lib/style.css";
// start at id = 0
let nextId = 0;
const comments = ref([]);
const load = async ($state) => {
  const response = await fetch(
    `http://localhost:3001/comments/next/${nextId}`
  );
  const json = await response.json();
  comments.value.push(...json);
  $state.loaded();
  // get elements created after the
  // last element returned in the current request
  nextId = json[json.length - 1].id;
};
</script>

Lastly, let’s see it in action.

With cursor-based pagination, your application will perform better even with larger dataset. Also, this concept has been around for quite a long time so there are a lot of more standard ways of implementing it depending on your needs.

Watch out for our next blog as we discuss how to optimize the vue app to handle very large comments as the user scrolls through more and more pages.

Thank you.