Paging for Non-Paginated backend

Now and then we encounter a backend that hasn’t been paginated. Usually, you can get in touch with the backend devs and ask them to implement paging. But sometimes that is not possible, so you would need to deal with paging on the client side. I recently encountered this problem while working on my Android app to show XKCD comics.

The XKCD API only allows you to fetch one comic at a time. So, if you want to show the comics in a list, you would have to fetch them one at a time. But there are almost 3000 comics. Retrieving all of them at once will cause memory issues for the app. This is why you only want to fetch a few comics at a time.

For my Android app, I am using Jetpack Compose with Paging3 to retrieve a few items at a time from a backend. I also implement caching so that the comics are available when I am offline. There is an excellent tutorial related to this from Phillip Lackner. I will only focus on the client-side pagination here.

Below is the method I call to retrieve a comic given its page number:

@GET("{comicNum}/info.0.json")
suspend fun getComic(@Path("comicNum") comicNum: Int): Comic

The received JSON is mapped to the Comic data class:

@Entity
data class Comic(
  val alt: String,
  val day: String,
  val img: String,
  val link: String,
  val month: String,
  val news: String,
  @PrimaryKey
  val num: Int,
  @Json(name = "safe_title")
  val safeTitle: String,
  val title: String,
  val transcript: String,
  val year: String
)

The getComic method needs to be called multiple times based on how many comics we wish to show for a given page:

val comic1 = api.getComic(1)
val comic2 = api.getComic(2)
val comic3 = api.getComic(3)
val comic4 = api.getComic(4)
val comic5 = api.getComic(5)

In the above, we want to retrieve the first 5 comics. But these calls are made asynchronously. So, we must wait for all 5 calls to finish before proceeding with the retrieved comics. How do we do that? We use Kotlin’s async lambda

val deferredComic1 = async { api.getComic(1) }
val deferredComic2 = async { api.getComic(2) }
val deferredComic3 = async { api.getComic(3) }
val deferredComic4 = async { api.getComic(4) }
val deferredComic5 = async { api.getComic(5) }

And then, we simply wait for all 5 deferred calls to complete before proceeding

val comics = mutableListOf<Comic>()
listOf(
  deferredComic1,
  deferredComic2,
  deferredComic3,
  deferredComic4,
  deferredComic5
).forEach { deferredComic ->
  comics.add(deferredComic.await()) 
}

This shows us how to get the first 5 comics. But what if we wanted to make it more generic? Meaning, we retrieve the comics for any page given that each page must have a fixed amount of comics (like 5 in our case).

Ascending order

In the XKCD example, the comic with the smallest index is 1 and we will also number our pages starting from 1.

suspend fun getComics(page: Int, pageCount: Int) = withContext(Dispatchers.IO) {
  // Code here
}

For each page, we must know the number of the first comic and the number of the last comic.

suspend fun getComics(page: Int, pageCount: Int) = withContext(Dispatchers.IO) {
  // page 1: (1 - 1) * 5 + 1 = 1
  // page 2: (2 - 1) * 5 + 1 = 6
  // page 3: (3 - 1) * 5 + 1 = 11
  val numFirstComicOnPage = (page - 1) * pageCount + 1
  // page 1: 1 + 5 - 1 = 5
  // page 2: 6 + 5 - 1 = 10
  // page 3: 11 + 5 - 1 = 15
  val numLastComicOnPage = numFirstComicOnPage + pageCount - 1
  // ...
}

We then need to make multiple calls to the back end in a for loop and store each deferred call in a list:

val comics = mutableListOf<Comic>()
val deferredCalls = mutableListOf<Deferred<Comic>>()
for (comicNum in numFirstComicOnPage .. numLastComicOnPage) {
  deferredCalls.add(async { api.getComic(comicNum) })
}

Finally, we create our comic list after waiting for all API calls to return the Comic JSON

deferredCalls.forEach { deferredCall ->
  comics.add(deferredCall.await())
}

Below is the full code:

suspend fun getComics(page: Int, pageCount: Int) = withContext(Dispatchers.IO) {
  val numFirstComicOnPage = (page - 1) * pageCount + 1
  val numLastComicOnPage = numFirstComicOnPage + pageCount - 1
  val comics = mutableListOf<Comic>()
  val deferredCalls = mutableListOf<Deferred<Comic>>()
  for (comicNum in numFirstComicOnPage .. numLastComicOnPage) {
    deferredCalls.add(async { api.getComic(comicNum) })
  }
  deferredCalls.forEach { deferredCall ->
    comics.add(deferredCall.await())
  }
  return@withContext comics
}

Descending order

You can modify this code to retrieve comics starting from the latest one down to the earliest one. For that, you would first need to retrieve the latest comic num and then subtract the page size instead of adding it. As of writing this blog, the latest comic number is 2904:

suspend fun getComicsInReverse(page: Int, pageCount: Int) = withContext(Dispatchers.IO) {
  val latestComicDeferredCall = async { api.getLatestComic() }
  val latestComicNum = latestComicDeferredCall.await().num
  // page 1: 2904 - (1 - 1) * 5 = 2904
  // page 2: 2904 - (2 - 1) * 5 = 2899
  // page 3: 2904 - (3 - 1) * 5 = 2894
  val numFirstComicOnPage = latestComicNum - (page - 1) * pageCount
  // page 1: 2904 - 5 + 1 = 2900
  // page 2: 2899 - 5 + 1 = 2895
  // page 3: 2894 - 5 + 1 = 2890
  val numLastComicOnPage = numFirstComicOnPage - pageCount + 1
  val comics = mutableListOf<Comic>()
  val deferredCalls = mutableListOf<Deferred<Comic>>()
  for (comicNum in numFirstComicOnPage .. numLastComicOnPage) {
    deferredCalls.add(async { api.getComic(comicNum) })
  }
  deferredCalls.forEach { deferredCall ->
    comics.add(deferredCall.await())
  }
  return@withContext comics
}

You can modify the code to only retrieve the latest comic number when the app launches and then use that for populating the list as the user scrolls down.

Random order

What about retrieving and paginated comics but then displaying them in a randomized order. For that, we can get creative. First, get the latest comic number and then create a list of comic numbers starting from 1 and going all the way to 2904 (the latest comic number as of today):

suspend fun getComicsInRandomOrder(page: Int, pageCount: Int) = withContext(Dispatchers.IO) {
  val latestComicDeferredCall = async { api.getLatestComic() }
  val latestComicNum = latestComicDeferredCall.await().num
  val listOfAllComicNumsRandomized = (1..latestComicNum).toList().shuffle()
  // ...
}

Now, the list contains all the comic numbers we can use to retrieve comics in a random order. We can traverse the list from the first index using the same logic as when we were retrieving comics in ascending order of their number:

suspend fun getComics(page: Int, pageCount: Int) = withContext(Dispatchers.IO) {
  val latestComicDeferredCall = async { api.getLatestComic() }
  val latestComicNum = latestComicDeferredCall.await().num
  val listOfAllComicNumsRandomized = (1..latestComicNum).toList().shuffle()
  // page 1: (1 - 1) * 5 = 0
  // page 2: (2 - 1) * 5 = 5
  // page 3: (3 - 1) * 5 = 10
  val firstIndexForPage = (page - 1) * pageCount
  // page 1: 0 + 5 - 1 = 5
  // page 2: 5 + 5 - 1 = 9
  // page 3: 5 + 10 - 1 = 14
  val lastIndexForPage = firstIndexForPage + pageCount - 1
  val comics = mutableListOf<Comic>()
  val deferredCalls = mutableListOf<Deferred<Comic>>()
  for (index in firstIndexForPage .. lastIndexForPage) {
    val comicNum = listOfAllComicNumsRandomized[index]
    deferredCalls.add(async { api.getComic(comicNum) })
  }
  deferredCalls.forEach { deferredCall ->
    comics.add(deferredCall.await())
  }
  return@withContext comics
}

The above can also be used to retrieve comics in ascending and descending order (for ascending, simply don’t shuffle the list while, for descending, reverse the list). Hopefully, this can be used by you in your Android app too. Thanks for reading!