koa-pageable: Spring-style pagination in javascript

A JVM Developer’s journey into the modern js world

After working in the JVM / Spring ecosystem for over a decade, I recently worked on a javascript / nodejs project building a RESTful service application using koa.

Here are a few of my high-level takeaways:

• Wow, Javascript has improved significantly since I last used it a decade ago
• Flow types really go a long way to improving code correctness and refactorability
• Huge YES to async / await
• The npm / yarn, babel, webpack stack make maven look pretty good
• I really liked the middleware approach. They’re like much more usable servlet filters
• The overall node ecosystem appears to be really lacking in well-defined patterns for standard “Enterprise” (aka large multi-developer application) system Quality Attributes (non-functionals)
• Just a couple examples of “enterprise” level features I found lacking:
  - Logging —there appears to be no straightforward way to output which file a log statement originated in (other than hardcoding). No standardized packaging means no ability to set log level by package. No MDC.
  - Error Handling — many libraries don’t even bother declaring their own error types, they just throw Error instances even for well-defined cases. This means that error handling often involves parsing out message strings
  - Documentation — There is no good solution for generating documentation from JSDoc annotated source code. The most common jsdoc, esdoc, documentation.js all have limits and support different subsets of the language and features (e.g. lacking basic es6 support, improper / non-configurable class inheritance handling). It surprised me that there is no frequently updated project in this space.

On the whole though, I really enjoyed the experience. Personally, I’m still more comfortable with the many benefits provided by a statically typed language (say, kotlin) along with a framework that provides most “table-stakes” functionality out of the box (e.g. Spring Boot). That said, I certainly see value in the node ecosystem and, given the right use case, would be glad to develop on it in the future.

I couldn’t find it, so I built it

One Spring nicety that I immediately missed was Spring Data’s Pagination support. I looked at a number of the available libraries but while they all provided the base pagination functionality, they did not support all of the use cases I was used to seeing from my pagination library.

Desired features:

• Support offset (page number * page size = offset) based pagination
• Use query parameters (not headers)
• Allow client to request the response to be Sorted
• Provide a consistent response format for all paginated requests including the actual returned data and pagination metadata such as current page number, current page size, total number of pages, total number of elements etc.
• Provide flow definitions for strong typing

So, as I was unable to find a library that met the requirements, I decided to build my own leveraging a number of patterns that I’d found very useful in Spring.

While building out the library, we identified some additional desirable features. One of the consumers of our API is a React Native application using redux for state management. The front end team identified that it would also be useful to allow the api client to control the format of the data. While the standard format is, of course, an ordered list, front end applications using a state management system like redux, often normalize the data coming in, storing entities in a map-like data structure. We don’t necessarily want to solve that problem or become overly opinionated, but we can provide a quick win for redux-like consumers of an api if there is the ability to return data from the api represented as a map instead of an array.

koa-pageable

koa-pageable is middleware for pagination in Koa inspired by Spring Data's Pagination support.

It provides control to the clients of your API, allowing them to specify the amount, order, and format of the data they can retrieve.

If you’re familiar with Spring Data then many of the same domain terms will be familiar to you:Pageable, Page, Sort. It goes a step further than Spring Data in that it also allows the client to partially specify the structure of the returned data.

Let’s pretend that you have a RESTful endpoint /people that is backed by a datastore containing 1000 people. If a client wanted to retrieve a list of people in batches of 10, sorted by lastname, they could issue the below request to get the second such page (pages are 0-indexed):
GET /people?page=1&size=10&sort=lastname

Usage

Note: examples below include flow types as I’m a big fan of static typing. These are definitely optional though.

First install the library via your favorite dependency manager, I like yarn

yarn add @panderalabs/koa-pageable

Then add the middleware function, and pass the pageable instance that it populates into ctx.state.pageable to your data tier:

Finally, you must ensure that your data tier properly implements the client’s intention (sorting the request, limiting the data size). The exact mechanism for doing this will depend on your data-access/ORM framework, but this is what it looks like using objection:

There is only one other concept to be aware of, and it relates to the format of the data serialized back to the client.

Array or Index

koa-pageable can serialize the result of a request in two different formats depending on the client’s request (value of the indexed query parameter).

Array format
Content returned in a simple ordered array.

GET /people?page=1&size=2&sort=firstname,lastname:desc&indexed=false

Indexed format

Returned as an ordered array of ids, and a corresponding index which is a map of {id: content item}. Note: in order to automatically return data in indexed format the underlying data type (in this case the person class) must have a property named id, the values of which must be unique within the data set.

GET /people?page=1&size=2&sort=firstname,lastname:desc&indexed=true

That’s it! That’s everything you need to know to bring some Spring-inspired paginated goodness to your koa nodejs app!

For more detailed information you can checkout the project readme on github and the API Docs.

We’ve already used this framework on a number of projects, and plan to continue supporting it in the future.

I’d love input, feedback, collaboration on the library and welcome any Issues and/or Pull Requests.

Me and my good buddy Domino, hard at work

At Pandera Labs, we’re always exploring new ways to build products and iterate on our engineering processes, and we value sharing our findings with the broader community as our company and our technology evolve together. To reach out directly about the topic of this article or to discuss our offerings, visit us at panderalabs.com.