A Plan for Data-Fetching

The quandary

When learning how to build apps using Flux-inspired architectures like React+Redux or Elm, this is one of the first questions people ask:

How does data-fetching fit in?

The received wisdom is to overload action creators to perform side effects, which are then called from lifecycle methods, but this is problematic.

So what’s the problem?

To illustrate, consider a situation where you want to show more content on wider screens, in order to take advantage of the extra real estate. You might take this approach:

  • In componentDidMount, only fetch the extra data if the viewport is sufficiently wide.
  • In render, only render the extra data if the viewport is sufficiently wide.

For example, consider this code:

After deploying this to your users, you quickly realize it doesn’t handle cases where the user resizes the viewport after componentDidMount fired. Wrapping our side-effect in an action creator did nothing to address the crux of the problem: when should a side-effect be called?

I don’t want to think about the mess of conditional code necessary to make such calls at appropriate times and without redundancy, once you factor in viewport resizing and real-time updates.

Can we finally just admit side-effects are bad?

Mainstream efforts to sanely model side effects in an architecture that wants to be purely functional (i.e. React/Redux) are trying to rationalize an anti-pattern. The libraries, tutorials and habits growing up around this trend are holding back the React community from reaching its full potential. We need a better way.

The vacancy observer pattern

My solution is something I call the vacancy-observer pattern. Its goal is to allow you to write all your React components in the functional style.

But before continuing, I want to take a step back and look at how functional programming fits into the world. Gary Bernhardt covers this in his boundaries talk, but in a nutshell, a real-world program consists of one or more functional cores, embedded in a matrix of non-functional (i.e. mutable, stateful, imperative, etc.) logic which connects them together and to the rest of the world via a message-passing strategy. Here, “rest of the world” includes databases, remote services, CSS, the DOM, the React engine, and even other logic modules in your app.

So for example, rendering a button might have the “side effect” of causing the user to click it, but we don’t consider that a violation of the functional style, because the thing that’s affected—the user—lives outside our island of functional purity. It’s the same with vacancy observers. What we render affects the vacancy observer, but it isn’t a side effect in a way that violates the functional style, since the observer isn’t part of our functional core.

So what are vacancies?

I’ll start with the basic mechanisms. A vacancy is a machine-readable hint embedded in the DOM about a slot in the UI where you don’t have the data you need in order to render. For example, if you’re rendering the product detail page, but no product details exist in memory, you’d render a vacancy in the form of a data-vacancy attribute.

These hints are in turn visible to a vacancy observer: a mutation observer running in the background which monitors changes in the DOM. When a vacancy appears, it performs a data fetch, feeding back into the reducer with the results.

From the perspective of the component writer, it becomes a matter of simply rendering a vacancy when no data is available to fill a given slot. Beyond that, it isn’t your job to worry about how data-fetching happens.

Code example

Here’s a reformulation of the previous example using this pattern. Notice that the componentDidMount method completely goes away, allowing our component to be expressed as a pure function:

Meanwhile, in a separate part of the app, our vacancy observer library exposes an API similar to a URL router. Its job is to declare how vacancies translate into actions that feed into Redux. This code runs once at app startup:

Now, if the viewport is widened above the threshold, the Related Product pane appears, which adds a vacancy to the DOM. The observer responds to this, and a network call is made. Upon success, the data we need becomes available to the component, which re-renders and the vacancy disappears.

How is this purely functional?

So that outlines the basic mechanisms in play, but I anticipate some questions. First, you might wonder, how is this not a violation of the functional style? Isn’t this tantamount to performing data-fetches from within the render function, which would be the most egregious possible example of a side effect?

The litmus test is whether you’d need to mock network calls in order to unit test. In that light, no, a component that renders vacancies remains a pure function of state; all you need for unit testing is to check whether the component renders a vacancy on a given set of props. The mutation observer is only instantiated in the running app.

Wouldn’t this create redundant fetches?

With React components, rendering is cheap and frequent. React is nice because we don’t have to obsess over when or how often rendering happens. But then, suppose our component produced a data vacancy three times in rapid succession. Wouldn’t that cause three duplicate network calls to be made?

Remember that React doesn’t disturb DOM nodes if nothing changes. Those three consecutive data vacancies would look, to the observer, like a single data vacancy which persists for X amount of time, resulting in a single call.

Just to press the issue though, even if you go out of your way to spuriously trigger the observer, it’s trivial to de-dupe redundant in-flight requests at the library level, similar to how React itself avoids redundant DOM mutations using clever diffing and comparison techniques.

What benefit does this have?

The benefits of this are mainly just the benefits of functional programming, which are fairly well-established, but here are a few highlights from my own experience using this technique:

  • Components are expressed as pure functions.
  • No more wrapper components whose only job is to encapsulate “when do I call my side effects?” logic.
  • No need for lifecycle methods.
  • Adding/removing things from the view implicitly and automatically adds/removes the corresponding network calls. This applies whether we’re talking about adding/deleting code, or dynamically showing/hiding various sections of UI.
  • Easy to unit test whether a component renders a vacancy on a given set of props.
  • No need to stub in side effects when writing unit tests for components.

An MIT-licensed implementation

What would an implementation look like, you ask? I have a library available on GitHub, which I’m currently using to great effect in an app of medium (and growing) complexity.