Choose the Right Pagination for API

Photo by Anastasia Zhenina on Unsplash

When designing an API for retrieving lists, one of the first things we think about is pagination. Without it, an overly extensive list can result in a request that’s not only time-consuming but also strains both the client and the server, potentially even causing the system to crash due to the sheer volume of data.

In this article, we’ll explore the two most popular types of pagination: offset pagination and token pagination.

Offset pagination

In this type of pagination, the client sends a request specifying the page number and the number of items per page. Ultimately, this translates into an SQL query using limit and offset.

The response usually also contains the total number of elements on all pages.

Pros

1. Easy to implement
2. Perfect when you need to display a table on the UI

Cons

1. The number of items must be finite.
2. SQL queries for counts(*) can be quite slow if preliminary calculations aren’t made.
3. Doesn’t allow for pagination across multiple sources. For instance, if we have two tables, booking_logs and order_logs, and we want to display them on the UI as a single list.
4. When navigating between pages, the same items may appear again if new records are being created during that time.

Token pagination

Token pagination can be implemented in different ways. I’ll show you one of them.

In the initial request, we send a typical search query. The response contains the token to retrieve the next portion of data.

In following requests, we send only the next token.

To avoid storing unnecessary states on the backend, we can embed the search query and the ID of the last returned item into a token. We can then compress the data using gzip or snappy and convert it, for instance, to base62.

Token pagination from different sources

Let’s say we have multiple sources: booking_logs and order_logs, and we need to return combined data from both sources.

With token pagination, we can embed the last returned ID from each source into the token. When executing a request for 3 items, we select 3 items from each source, and then sort them on the backend to determine which specific items to return.

New last returned IDs will be contained in the next token.

Pros

1. Perfect for infinite lists.
2. Allows for pagination across multiple sources.
3. Each item will be displayed no more than once.

Cons

1. More complex to implement.
2. Doesn’t show the total number of items.

Conclusion

There are many ways to do pagination, and it’s important to pick the right one for your specific task. There isn’t always a single best solution; it’s all about understanding your needs and choosing wisely.