Advanced Redux Entity Normalization

One of the earliest problems a developer hits when dealing with Redux store organization is arrays of entities are difficult and inefficient to work with (both computationally and developer experience-wise). The official Redux documentation provides a very good primer on what the best way to normalize a store looks like. An astute reader will also note similarities in shape to other solutions like the normalizr library.

While it may feel counter-intuitive to structure data like this, we must remember that Javascript doesn’t actually guarantee order of keys in an object. In addition, insertions, replacements, and removals (especially in the context of maintaining immutability, which in Javascript requires cloning) is significantly more efficient with Objects than Arrays.

This is all fairly standard for initiated Redux users. Where the documentation doesn’t explore is the problems when 2 independent views of the same, de-duplicated entity must exist at the same time and independent meta-request information like fetching status, pagination status, must be tracked independently.

At Grovo we have been using an abstract structure we’ve taken to calling Key Windows.

The concept is fairly simple, and in a way if we’re thinking of normalization as SQL tables, we could think of this as a ‘view’. For a given entity type there are multiple possible lenses to view the data: a paginated table, a type-ahead dropdown, a detail view, a special widget. We consider these lenses a ‘window’ into the raw de-duplicated universe of entities.

The actual structure is extremely simple and an extension of the official documentations suggestion, merely replacing the allIds with an arbitrary number of meta-information containers and a keyWindows top-level tracking all currently present windows.

Where the power comes in is the recognition that there is a very strong attachment between key windows (“what am I looking at”) and API requests. It makes sense to begin to attach all sorts of meta-information about requests and views of data inside the key window data structure like fetching status and pagination.

Starting with a trivial example: I’m viewing a paginated list of Users, but our UX calls for a little widget that shows a list of “managers” at the top. Because the managers potentially are not currently in the page, we cannot simply replace the byId portion of the store lest we accidentally wipe out our managers, and we cannot simply show all users in byId in the table lest we show incorrect information. Our store would then look something like the following:

This structure gives us a number of benefits:

  • We can track request/fetching status of the browse users table and the managers widget independently
  • We can track the pagination status of the browseUsers independently as well, potentially allowing a single view to have multiple paginated data sets
  • Maintenance of the byId key on the store is a simple concat of new entities on request complete
  • The ids array in each key window is a simple replace on request complete
  • The ids array continues to maintain any intrinsic ordering received from the API endpoints

Implementation Notes

We have found that fully adopting this structure required a few changes to our actions/action creators, namely we now need the entity responsible for dispatching a given action also needs to explicitly declare a key window name. In the above example, fetchUsers({query}) might soon become fetchUsers({query}, 'browseUsers'). However “explicitly declare” can also mean that something like fetchManagers() can always use 'allManagers' internally. Simply put the action payload needs an explicit window name.

Because the keyWindow property on the action is both common and explicit even on the subsequent ‘receive’ action(s), it in effect tags an entire request lifecycle and simplifies our reducer allowing two supposedly distinct fetch initiation actions to share a common implementation.

One addition item to note, this relies on your side effect system emitting the receiveEntities() action to namespace the entities behind their entity name (see lines 24 and 30) above, however you can always still do a simpler receieveUsers() action. I personally have just become a fan of a singular receive action for multiple fetch initiators.

Further Improvements

There are of course ways we can build on this concept:

  1. Tracking and handling errors can now use the the key windows to isolate and provide granular error reporting and recovery
  2. Tracking the unique list of key window names seems like overkill, but allows for you to deterministically loop through all key windows and allow for garbage collection via a releaseKeyWindow() type action
  3. Since key window names can be anything, they can be another entity’s ID to provide some semblance of foreign key type relationship for quicker lookups

A Word on Naming

We internally called this concept “key windows” after a period of debate that was longer than I care to admit. We initially called them views (like SQL views) but found the word view is used quite often in frontend work (who knew) and caused lots of communication problems and unclear messages. My problem with key windows is it doesn’t fully explain what it is in the name itself, but I’m not sure if anything better exists. I’m open to suggestions!

Naming really is the worst problem in computer science…