Firestore pagination with real-time updates on Android

When using Firebase Firestore, we can listen to real-time updates by using addSnapshotListener . This would listen to any change in your record and notify you so you can update your UI accordingly. This can work in small data sets but you have a large one, you’re going to have to use pagination but when you use pagination, you can’t opt-in to real-time updates.

In this article, I’ll explain my workaround that gives us both real-time updates and pagination at the same time. We’ll be using Android Architecture Component’s Paging Library for efficient pagination. We’ll also be using RxJava2. Personally, I use Clean Architecture but to keep things simple, we’ll be using MVVM with a Repository pattern as our app architecture.

As an example, I created a Firebase project that just has records containing a name and active property.

We want to:

• Enable Pagination by retrieving in batches of 10 records
• Get Real-time Updates by listening to the active property and changing the UI accordingly

Full source code can be found here

Manual Pagination

If you’ve ever tried pagination using Firebase Firestore, you might have used queries this way:

So we have a repository that has a getRecords function so we can get the items from the database paginated at 25 records per retrieval and a refresh function to reset the pagination.

This could work but the implementation is kind of bad because we have to do some sort of state management. The function needs to know if it’s getting the record for the first time or not but for the meantime, let’s still use this anyway.

Now we can use it like this:

Manual Pagination

This would now give you pagination for your list so if you have a large amount of data, those data would be loaded in chunks and your app won’t have to wait until all of those data get loaded but by using pagination, we’re letting go of realtime updates. That means, we won’t know if a record is removed, updated, or added. We’d have to manually refresh the list so it would get the data again.

FirestorePagingAdapter

A cleaner solution here would be to use the FirestorePagingAdapter provided from FirebaseUI.

With this we won’t be needing the repository and just use the adapter directly like this:

Pagination with FirestorePagingAdapter

Now, this is much easier to implement. Pagination is going to be available without a lot of effort. But again like the manual solution above, we’d have to manually refresh the list so we would get updates.

However, it removes the MVVM aspect of our code. The Firestore Query should be in the data layer but you have to put it here because the adapter needs it. You’re able to touch the data layer-related code in the presentation layer and that removes the idea of having an MVVM Architecture along with the Repository Pattern.

Pagination with Real-time Updates

So in my app, I wanted to have to those two things:

1. Real-time Updates with Pagination
2. Clear Separation of Concerns between layers

Luckily, I was able to come up with a solution but of course, there had to be some work.

First, I created a wrapper for my data class like this:

It just contains the id so we can use that for the PagedListAdapter later, and then the Item wrapped in an Observable so we can receive streams of updates later on

Let’s go back to the repository:

Now, my repository uses Kotlin Coroutines so I can have nice imperative code. I also used Kotlin Coroutines for Play Services so I can use await instead of the addOnSuccessListener callback

So what’s happening here?

I added a loadBefore and loadAfter parameter which accepts the string ID of the boundary item. Both of them are nullable but only one of them should be specified. When one of them is specified, we just get that item so we can get the DocumentSnapshot and add that snapshot to the query using endBefore or startAfter. If they’re not specified, we just execute the query as is and it would get the initial load

Next, once the queries have been adjusted as needed, we get the value and once we have it, we map the object to our real-time data class.

I added a getItem function which creates an observable using Firestore’s addSnapshotListener then emits that value for its consumers.

Now, every code related to fetching the item is contained in our repository and the presentation would just have to call it.

In our presentation layer, we’d have to create a DataSource for the paging library. There are 3 different types of DataSource depending on the type of pagination you’re having:

1. ItemKeyedDataSource — Using an ID and getting items before or after that ID
2. PositionalDataSource — Using the current position and getting items before or after that position
3. PageKeyedDataSource — Using a page number. Items from page 1, page 2, …, page N

For Firestore, we’re going to use ItemKeyedDataSource since we only have startAfter and endBefore queries for pagination so here’s how our class should look like:

The loadAfter and loadBefore functions are very convenient as they handle when we have to load and what ID should we use for loading through LoadParams . We just use those and pass that to our repository function.

Now, let’s use this in our ViewModel:

There’s so much boilerplate with for coroutines so if you’re using the 2.1.0-alpha04 for the Lifecycle library, use the viewModelScope extension they provided.

That’s that for the ViewModel and now, our Activity can go back to first Activity I’ve shown you minus the code for scrolling, refreshing and loading the initial items

Much cleaner now!

Lastly, we have to make a few changes to our ListAdapter by changing it to a PagedListAdapter and observing the observable for real-time updates.

Realtime updates with pagination!

Now, we have both pagination and real-time updates to our records!

Take note of on onViewRecycled function as it would handle clearing the subscription when the view gets off the screen. This is so that only the views inside the screen would get real-time updates.

Limitation

Now, yes, this solution can give you updates to your paginated list however, it can only give you updates on record changes and record deletions. This won’t give you updates on new records. You’d have to manually refresh the list or go up and down so the Paging library would re-query them. To me, this is okay since I just need the updates to be seen on the list and I won’t have records added or deleted frequently.

This is just a workaround that I figured out because I wanted to have clean code and needed the real-time updates on my paginated list. Only do this when you really need real-time updates on a very large dataset. If you have a small dataset, you can just use the addSnapshotListener directly. That gives you real-time updates on your collection but if you have a large dataset, in the meantime, maybe you can use this solution as it worked for me. Let’s hope that Firestore would enable pagination with real-time updates soon.

If you have any questions or suggestions, please leave a comment below or contact me and I’d be happy to attend to them or improve this solution!