Infinite Scroll’ing the right way

Scroll — The basic:

Scrolling is an implementation of data display whenever we cannot fit everything onto a single page. Most websites are generally implemented this way due to the inability to fit all the content on the computer screen (Twitter, Facebook feed… etc).

The problem:

To simply implement this, we can always put all the data on the screen. The problem arises however when the number of items becomes large. Again, we can look at this Twitter feed below.

When we encounter a large list like this, displaying all possible items on the screen is not at all feasible since the number can be hundreds, thousands… or more and making that many DOM nodes to display data is simply not an optimal solution.

The Solution:

One of the solutions for this is using Pagination.

We organize our documents by pages. Each page will hold a certain number of items and display them as the user decides to navigate forward and back, using either page number or the arrow icon, just as flipping through the pages in a manual.

In the technical perspective, this makes sense. We organize items by content page in our database pages and depending on the user’s input, we can serve out the correct data.

But how can we solve this issue in a more efficient manner, in a more user-friendly perspective? What if we dynamically change the pages for the user as they scroll?

With Infinite Scroll, we can solve this issue by dynamically appending the pages as the user scrolls, incrementing the page one at a time until we run through all the pages. With this approach, we can reduce interaction cost, especially on mobile by saving extra clicks from users.

Implementation:

We could add an eventListener on scroll to check whenever we scroll down to the bottom of the page, then start loading more items for our infinite scroll list.

window.addEventListener('scroll', function(e) {
  // debounce scroll check
// do something
})

Doing this however can be both noisy and lead to non-performant Javascript code due to the nature of the event listeners. When trying to optimize the front-end code, our objective should be to reduce the number of listeners whenever possible.

Enter: Intersection Observer

A colleague of mine recently wrote this article https://medium.com/walmartlabs/lazy-loading-images-intersectionobserver-8c5bff730920 and the basis of this API can be used to implement Infinite Scroll properly.

Without going too deeply into it (the details can be found on the post above), we essentially can “asynchronously observe changes in the intersection of a target element with an ancestor element or with a top-level document's viewport” (taken directly from https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API).

The biggest improvement of this API over the use of an event listener is that the computation of both the target elements and the actual observing behavior of the intersecting elements are not run on the main JS thread, thus freeing us from potential noise and blocking nature of the language.

entry.boundingClientRect
entry.intersectionRatio
entry.intersectionRect
entry.isIntersecting
entry.rootBounds
entry.target
entry.time

With this info available through the callback, how can we compute the scroll position to handle Infinite Scroll properly? This post here has an amazing summary of direction detection for observed elements.

Things to consider:

We need to keep the count of the visible DOM elements to the minimum. This should be however larger than the viewport (our root element). Through some testing, 20 seems to be a good number for this, since it will encapsulate a size larger than the viewport on both web and mobile.

We will get familiar with a few keywords associated with building an infinite scroll feature and I will delve into each.

  1. Root Element
  2. Sentinel
  3. Recycled DOM nodes
  4. Sliding windows

The Root Element is the element that we will use to observe our sentinels. Whenever the sentinels intersect with this element, we will handle the computation for either fetching new items via an API or grabbing it directly from the cache. This keyword comes directly from the Intersection Observer API. We extend the root margin out on the bottom to ensure the fetch gets called way before it comes to the viewport, thus reducing the amount of time the user has to wait for the new data to come through.

The Sentinels are our anchored elements that can be used to trigger the changes. For an infinite scroll, we use the top and bottom anchors to detect changes, whenever they intersect with our Root Element. The top sentinel is simple. We simply return the data that we save in our cache (in memory, Redis map… etc) and display. The bottom sentinel is a bit trickier since depending on our sliding window, we need to either make a network request for more data or grabbing from the existing cache (user scrolls up then back down). In our demo, we will simplify this by having all data existing already in the cache, but keep in mind the necessity of having the fetch call whenever data from the cache runs out.

Recycle DOM nodes is important since we do not want to repaint the entire browser to display new items. By changing their attributes, we can display new data without forcing a full re-render. By carefully extending the paddings of the container, we can emulate the list being extended to the user without actually extending it in the DOM. I will go in depth about this implementation in the demo.

Sliding Windows refers to the areas between two indices when we scroll up or down, equal to the amount of displayed DOM nodes. By having this sliding window, we only need to display a fixed number of data while keeping the rest away from the actual rendered page.

With these out, let's go through the actual demo.

As we can see here, the number of DOM elements are fixed (20 in this case) as we scroll through the entire list. We dynamically change the attributes and contents of these elements as well as the paddings to emulate a large scrolling list.

A few functions to keep an eye out for

  1. initIntersectionObserver
  2. recycleDOM
  3. adjustPaddings

The initIntersectionObserver method initializes the observer and wire all the necessary components for this to work. Our options can be left blank or passing in the root, rootMargin, and threshold (refer to the official doc here for more granular config depending on usage). The callback will run every time the sentinels intersect. In our case, we anchor the sentinels as our top and last rows of the list (line 17–18) and of course, our listSize is fully customizable to all situations.

The recycleDom method is used to recycle existing DOM nodes. Depending on needs, you might need to modify and might even need to force a new render on some elements. However, the main point to keep here is that we only replacing contents and not completely mounting, unmounting new elements every time the Sentinels hit the root element.

The adjustPaddings method is used to get new paddings and directly modify the container’s size. Why do we need to do this? Since we recycle our DOM nodes, the actual content is always stuck at a fixed number of elements. By extending the paddings on either direction, we can emulate the list extending larger or becoming smaller as we scroll up and down, by the same amount of the items’ heights being removed. In some cases, we might need to also compute and cache the heights of elements being removed to add paddings accordingly. For convenience in our demo, we set the height of each row to be 150pxand margin 10pxon top and bottom, thus we are removing and adding 170px * num of itemsbeing removed on scrolling (line 5).

Another way to handle this, as you might see in some implementations of Infinite Scroll, is to use Tombstone Elements. Basically, instead of extending the container’s size, we add empty elements to replace elements being removed. However, this implementation still clutters the DOM with elements and I find extending the paddings to be better.

Conclusion:

Making a performant infinite scroll feature is difficult. However, with Intersection Observer API, this can be done well and performant. In addition to infinite scroll, Intersection Observer can be used to lazy load images, run animations, and many other things. Feel free to try out the complete demo on jsFiddle below and let us know what you think!

Input parameters:

1. The number of items in the virtual DB

2. The number of displayed DOM elements.

Due to a strange bug with jsFiddle, I could only set the minimum of 20 DOM elements for this to work correctly. I suspect this is due to using Intersection Observer inside an iFrame causing the Root Element to not be computed properly. This number in production can be smaller than 20, as long as the two sentinels positions outside of the root (viewPort in this case).
For a more accurate view of the demo, please click the link below the embedded jsFiddle.

Link to jsFiddle


Here is the list of supported browsers for Intersection Observer. For the ones that do not have, here is the polyfill by W3C https://github.com/w3c/IntersectionObserver/tree/master/polyfill

Image taken from https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API