Building a Scalable Polyrepo using Node and React

Photo by Florian Klauer on Unsplash

As Unified has grown so has the number of supported use cases and therefore the complexity of our reporting and analytics platform. Maintaining and scaling our core frontend application has been a challenge and the engineering team has taken several approaches to ensure that it remain modular and performant despite its growth. In this post I’ll walk through one of the larger evolutions of Unified’s frontend and discuss how the project has changed to meet the needs of a growing application in a highly dynamic development environment. Specifically I’ll be covering the how and why of our decision to switch to a polyrepo for our platform’s frontend.

The Monorepo

The server side components of our frontend use Node while our client side components are built in React. The first version of our frontend wasn’t much more complex than this and all code lived in a single Git repo. This made sense given the number of contributors and size of the codebase. Everyone on the team could work against the same repository and share dependencies making the build process fairly straightforward. Below is a simplified example of the project structure assuming a single React app with a Node server.

.
├── apps
│   └── analytics-app
│       ├── actions
│       ├── appEntry.jsx
│       ├── components
│       ├── routes
│       └── stores
├── assets
│   ├── images
│   └── styles
├── dist
│   └── analytics-app
│       ├── app.js
│       └── main.css
├── node_modules
├── package.json
├── server
│   ├── index.js
│   ├── routes.js
│   └── templates
│       └── AppLayout.jsx
├── server.js
└── tests

The “apps” directory contained all our client side code broken into separate React apps while the server side code lived in the “server” directory. When building the Node project we could transpile and minify the appEntry.jsx file in each app to pull in all the required dependencies and create an app bundle. The bundle could then be copied to the “dist” directory as app.js along with any styles. When a user requested a page the Node server would route the request and then generate an HTML template with script and style references back to the React app.

Server Side Template

This initial approach was simple on the server side and on the client side it followed a similar standard pattern. Most of the UI was rendered from a single script tag containing the app code. Once a browser downloaded the HTML template and fetched the React app bundle it could then render the components in the bundle. A standard template component wrapped each application to enable common functionality such as navigation and any custom props could be passed down from a common config if needed. Dynamic global state, information about the user, API tokens etc., would persist in the Flux store and could also be passed down as props.

Client Side React App

While this was a good initial solution, as the number of smaller react apps increased and the size of the team grew it became increasingly difficult to maintain the entire application in a single repository. Not only did the codebase became more complex but because all node package dependencies were shared across each app, developers for different apps were locked into particular versions shared by the whole team.

From a maintenance perspective this became problematic if, for example, one team managing one app wanted to bump the React version. The speed at which the React ecosystem was evolving meant that newer version weren’t always backwards compatible and often there was a great deal of work required to support version changes from reimplementation to testing. If a team wanted to be able to take advantage of a new React feature they would have to take on version bumping the entire platform. As a result, every few months each team was sidetracked from feature development with arduous package version maintenance.

While there are certainly advantages to maintaining a monorepo for large enterprises with hundreds of developers the advantages didn’t apply at our scale and there was little appetite for investing in the kind of VCS that would eventually be necessary to support a massive code base.

Building the Polyrepo

In order to migrate our growing Node/React project to a polyrepo the team decided to move individual React apps into their own Git repositories and then package each one as a node module. Using this approach they could all easily be loaded into the platform via the package.json file of the Node project. Each app would have its own dependencies and different teams could develop against each app independently of one another. Whenever they wanted to release a new version of their app they would simply just bump the package version in the main Node project. In order to mount each app in the Node project however we had to make a couple of changes to the HTML template, the client side code, and the build process itself.

The first change was that all custom Unified node modules containing a React app came prebuilt and prepackaged into a lib.js file that could be loaded right before the container code in the app.js file. Therefore the HTML template was modified to look something like:

The first script tag contained the packaged version of the React app imported as a node module including all its components and dependencies etc. The second script tag contained code used to mount that child or library app. The container code worked by loading a lightweight React wrapper which called a function inside the specific library React app.

As you can see a LibraryApp component is declared which accepts a set of props specific to the custom React app being rendered. In that LibraryApp component a simple div is rendered and then, once mounted to the dom, the entire child app is rendered. App routing is set up in that app independently. This approach had the advantage of supporting a more gradual transition because it did not require that all child apps be broken out into their own repos all at the same time. Once the initial template was set up, however, migration was a fairly straightforward process. Each team could then manage their dependencies independently of one another and maintain their own processes around version control and development. When complete the final node project structure would then look more like:

.
├── apps
│   ├── analytics-app
│   │   └── appEntry.jsx
│   └── reporting-app
│       └── appEntry.jsx
├── dist
│   ├── analytics-app
│   │   ├── app.js
│   │   ├── lib.js
│   │   └── main.css
│   └── reporting-app
│       ├── app.js
│       ├── lib.js
│       └── main.css
├── server.js
├── node_modules/
│   └── @unified/
│       ├── analytics-app/
│       │   ├── dist/
│       │   │   ├── app.css
│       │   │   └── app.js
│       │   ├── node_modules/
│       │   ├── package.json
│       └── reporting-app/
│           ├── dist/
│           │   ├── app.css
│           │   └── app.js
│           ├── node_modules/
│           ├── package.json
├── package.json
├── scripts/
├── server/

As you can see the library javascript and styles come prepackaged in the “dist” folders of the Unified custom node modules and can be copied directly into the “dist” folder of the Node project. By breaking out the project into a polyrepo we were able to address some of the pressing issues facing the team however the new structure was far from a perfect solution. In subsequent posts I’ll be discussing how the team refined this approach to improve performance and reduce dependency redundancy.

If you’re a developer looking to join a team working on interesting problems at the intersection of technology and marketing be sure to check out Unified at https://unified.com/about/careers-and-culture.