DynamoDB Pagination

What is Pagination?

Pagination is splitting a database output into manageable chunks or pages. To do this:

1. Decide on the maximum number of database rows that can be included in each page. You may hard code this value, or you define it in a variable so that the value may be changed at runtime.
2. Inform the user that other ‘pages’ are available and provide a mechanism whereby the user is able to select a different ‘page’ of details. I currently use a set of hyperlinks in a separate pagination area which looks like:

Pagination area

What pagination support does DynamoDB provide?

Amazon DynamoDB documentation says that DynamoDB paginates the results from scan/query operations. With pagination, the scan results are divided into “pages” of data that are 1 MB in size (or less). An application processes the first page of results, then the second page, and so on.

A single scan will only return a result set that fits within the 1 MB size limit. To determine whether there are more results, and to retrieve them one page at a time, your application should do the following:

1. Examine the Scan result:

• If the result contains a LastEvaluatedKey element, proceed to step 2.
• If there is no LastEvaluatedKey in the results, then you are done. There are no more items to be retrieved.

2. Construct a new Scan request, with the same parameters as the previous one — but this time, take the LastEvaluatedKey value from step 1 and use it as the ExclusiveStartKey parameter in the new Scan request.

3. Run the new Scan request.

How pagination is done?

There are two approaches for the pagination to be done.

1st Approach — using scan

• On the webpage where pagination is supported, the browser sends the LastEvaluatedKey to the server.
• To access the first page, browser sends the LastEvaluatedKey as undefined. With this undefined key, server sends the first page data and also sends the LastEvaluatedKey for the next page.
• To access the next page, the browser sends back the LastEvaluatedKey and the same procedure continues.

const async = require("async");
const _ = require("underscore");
const AWS = require("aws-sdk");
AWS.config.update({ region: 'us-west-2' });

const docClient = new AWS.DynamoDB.DocumentClient();

var startKey = [];
var results = [];
var pages = 0;
async.doWhilst(
    //iteratee
    (callback)=>{
        let params = {
            TableName: 'td_notes_test',
            Limit: 3
        };

        if(!_.isEmpty(startKey)) {
            params.ExclusiveStartKey = startKey;
        }

        docClient.scan(params, (err, data)=>{
            if(err) {
                console.log(err);
                callback(err, {});
            } else {
                if(typeof data.LastEvaluatedKey !== 'undefined') {
                    startKey = data.LastEvaluatedKey;
                } else {
                    startKey = [];
                }

                if(!_.isEmpty(data.Items)){
                    results = _.union(results, data.Items);
                }

                pages++;

                callback(null, results);
            }
        });
    },

    //truth test
    ()=>{
        if(_.isEmpty(startKey)) {
            return false;
        } else {
            return true;
        }
    },

    //callback
    (err, data) => {
        if(err) {
            console.log(err);
        } else {
            console.log(data);
            console.log("Item Count", data.length);
            console.log("Pages", pages);
        }
    }
);

2nd approach — using Query

Incase pagination with particular attribute then query with particular Partition key its use the below method.

const AWS = require("aws-sdk");
AWS.config.update({ region: "ap-south-1" });
const TABLE_NAME = "write your table name";

const docClient = new AWS.DynamoDB.DocumentClient({
  sslEnabled: false,
  paramValidation: false,
  convertResponseTypes: false,
  convertEmptyValues: true,
});

let paginationDB = async () => {
  
   let queryParams = {
     TableName: TABLE_NAME,
     KeyConditionExpression: "#PK =  :PK",//Enter partitionKey 
     ExpressionAttributeNames: {
       "#PK": "id",// Enter partitionKey name
    },
     ExpressionAttributeValues: {
       ":PK": "123456", // Enter partitionKey value
    },
  };
  let result = [];
  let items;
  do {
    items = await docClient.query(queryParams).promise();
    items.Items.forEach((item) => result.push(item));
    queryParams.ExclusiveStartKey = items.LastEvaluatedKey;
  } while (typeof items.LastEvaluatedKey != "undefined");
     console.log("result", result);
     console.info("Available count size:", result.length);
};

paginationDB()
  .then((res) => {
    console.log(res);
  })
  .catch((err) => {
    console.log(err);
  });

Thank you for reading!