elm-paginate: An API Design Post Mortem

TL;DR — I made an Elm library called elm-paginate to encapsulate pagination concerns from the rest of your app. This article talks about some of the considerations and challenges I faced in creating a clean and safe API, and my choices in how I overcame them. Here is a fully-featured demo of the library, and the demo’s source. If you need pagination in Elm, try it out!

Pagination is not a particularly tricky feature, but it does have a few sticky points to consider. First and foremost, pagination is a view/UX concern to make long lists easier to browse. The data layer has no need to paginate, and in fact, pagination can get in the way of data manipulation.

Additionally, pagination behavior isn’t always clear:

1. What happens if you delete an item from a paginated list? Does the page displaying that item have one less item than all of the other pages, or do items from the following pages “slide” in to fill the gap? The later would maintain consistent page sizes, but may need to remove a page if the last page has only one item.
2. Similarly, what happens if you insert an item in the middle of a list, does it “push” the other items forward by one, creating an additional page if needed?
3. What if you are on the last page and you delete all of the items on that page? Do you “jump” to the previous page?

In my implementation, I chose what seemed to be a sensible standard, taking what would be the result of applying any transformation to the list and “re-paginating,” maintaining the current page whenever possible.

Take 1: Applying pagination directly

A paginated list is really just a segmented or partitioned list. So the simplest approach is to partition your list by the desired page size and save it in your model as a List (List a).

But how do you know which segment to show as the “active” page? You could store the current page index as a separate field in your model, but then you need to keep that in sync with your list if the pagination changes, and you open yourself to the possible error of having an index outside the bounds of your list’s length.

This is a known problem with a known solution — a zipper list, which binds the concept of a “focused” element with the rest of the list, and provides a way to traverse the list without going out of bounds (note how next returns a Maybe.)

However, zipper lists can’t be empty, so if you need that option (say you are loading your data by http) you’ll need a Maybe (Zipper (List a)), which is pretty ugly, messy to manipulate, and error-prone. Imagine how you would map or filter that, or worse, update, insert or remove elements. You would need to manually shuffle items around to maintain a consistent page size. You could make a function that takes a transformation function, does a toList, applies the function, then re-paginates, but that is a lot of unnecessary work and doesn’t make for a great API either.

All of that for a view concern that the data layer doesn’t even care about! Perhaps it would be better to store the list unadulterated, and do the pagination dynamically as you pass the list into the view, instead of storing the stateful pagination data.

Take 2: Separation of concerns with a “pager” (1.1.0)

The dynamic approach seems much cleaner, but requires storing the current page index, and all of the problems mentioned with that.

I encapsulated those issues by creating a new type to store the stateful page information as a Pager. You would store the pager in your model along side of your list. As long as your pager knew the length of your list and the desired page size, it could calculate the number of pages and track the current page’s index, offering helpful navigation functions like next, prev, first and last. I even made my own numeric equivalent of a zipper list for Int’s to ensure the index would always be between 1 and the total number of pages to avoid “out of bounds” errors (see elm-bounded-number).

This way, you could treat your list as normal, and when you were ready for pagination you could ask the pager for the correct “slice” of your list to display, along with a pager-view to show the other available pages.

The only problem was that you had to manually keep the pager in sync with your list. If you performed an operation that changed the length of the list, you needed to update the pager too, which was tedious, messy, and opened the door to out-of-sync errors again, and still mixed the concerns of your business logic with pagination mechanics.

Take 3: Total encapsulation (2.0.0)

The final piece to fully separating pagination from the rest of the app was to manage this syncing problem automatically by expanding the encapsulation a little further.

I got rid of the Pager type and changed to a PaginatedList a type wrapping not only the pagination information, but also the list. Now the model only stores the paginated list, which keeps the pagination data in sync automatically when the list changes. The question now becomes, how do you transform the list, since it is “hidden” in the opaque PaginatedList type?

I solved this by adding a map function, which takes a List a -> List a function and applies it to the list, returning a new PaginatedList a. Although this requires “lifting” over an additional layer of structure whenever you want to transform your list, it is a standard pattern and very clean with minimal impact to your calling code, especially when compared to the previous approach, plus it guarantees that the pagination data is always accurate, eradicating that source of bugs. You can use map to update the paginated list in your model, or dynamically when passing it to the view, which is useful for sorting and filtering prior to display.

With these changes, the library covers all of the pagination concerns with an API and usability I approve of. It should be ready for use in the wild.

Parametricity?

The only problem with PaginatedList a is that it only works with List's. What if someone wants to use it with something else, like an Array or Dict or even something crazy, like a LazyList? Or more practically, users will likely be wrapping their data in an opaque type.

I could modify the library to accept any data structure (Paginate.Custom.Paginated a) as long as the passed in data structure had a length and slice function. This is where typeclasses would be useful, but Elm does not support them for various reasons. The alternative is to require passing in those two functions as arguments to some of my API functions (I could only require them on initialization and store them in the model, but in general you wan to avoid storing non-seralizable data like functions in your model). This would work, but it gets a little messy, and makes me wonder if the benefits are worth it.

I’ve gone back and forth on this, and so far I have decided against it. Pagination by definition implies an order, which theList type seems most appropriate for compared toArray's and Dict's. As cool as it would be to paginate an infinite lazy sequence, I’m not sure how practical that would be. Opaque types are the biggest concern, but the opaque type can simple wrap a paginated list, which works nicely, as its own API would wrap the required maping discussed earlier. I realize users often useDict's inside their opaque types for performant lookups and simpler updates, but if pagination is a requirement, perhaps a List is better anyway, unless performance becomes an issue.

Perhaps in the future I’ll add parametricity if there is demand, but for now I am happy with the library as it is.

UPDATE: I went a head and added pagination for custom types. If you can supply a length and slice function, you can paginate your custom data type. I made a fun little demo of this by paginating infinite streams.

And that is how I designed the API for elm-paginate.