Lessons learned in 2 years with a JavaScript/React Monorepo
Things we learned from 2 years of maintaining a JavaScript/React Monorepo
Two years ago, our team moved all of our JavaScript applications into a single monorepo. We’ve had time to let the dust settle on our decision, and to hit a few walls along the way — we thought we’d share them with the world.
Why a monorepo ?
We have a lot of different SPAs here at Botify. Our product is a suite of different components and tools that have different uses for different customers. Our JavaScript team of around 10 Engineers maintains:
- 4 major SPAs
- 3 more minor web applications
- An internally-hosted Storybook
- JS SDKs
- a Chrome Extension
- PDF generators (node bundles)
- Embedded browser bundles
This is a lot of code, in a lot of different places. Moreover, some of these packages have a lot in common (the SPAs), and some are wildly different in function and in the artefacts produced (Chrome Extension, Embeds, SDKs).
Originally these projects were all separate git repos, with their own lint, build scripts, CI/CD, codestyle, and sometimes even technologies. This caused a lot of problems:
- Codebase is diverse, some legacy is hard to maintain or left behind
- Commonization of UI/CSS is difficult (building views for multiple apps)
- Code uniformity is worse
- Lint/Build/Deploy/Release scripts have to be managed separately
- CI/CD has to be managed separately
- PRs have to be managed in different repos
- Dependencies between packages require multiple PRs, with dependency changes
- Team is slower at scaling in size
Two years ago we decided to bring them all into a single monorepo, to try and solve these issues. Here’s what we’ve learned along the way.
How we monorepo ?
Our JS frontend monorepo’s top-level directory looks like this:
├── .nvmrc├── .prettierrc├── .sass-lint.yml├── .travis.yml├── .editorconfig├── .eslintignore├── .eslintrc.js├── .babelrc├── CONVENTIONS.md├── README.md├── apps/│ ├── common-style/│ ├── common-js/│ ├── chrome-extension/│ ├── botify-embeddable/│ ├── sdk-js/│ ├── app1/│ ├── app2/│ ├── …├── config/│ ├── apps.js│ ├── …├── package.json├── postcss.config.js├── script/│ ├── …├── webpack.config.js└── yarn.lock
Each of the directories in ./apps has its own package.json and dependencies, and each app produces a separate artefact versioned and released on its own. Some apps, like common-style and common-js, are just commonized code that is imported inside some of the applications, they do not produce any artefacts.
Sharing a common core
The biggest gain we’ve had from switching to a monorepo has been our increase in commonization, and uniformization of our code. As much as possible we try to reuse code and have common rules that are respected throughout the codebase. This allows every engineer to be able to work on any part of the codebase and feel at home.
Here are the things we’ve made common for our whole codebase:
Common rules
We have a set of rules and a code style, and we enforce it through CI (a PR fails CI build for style issues). This allows us to leave style discussions out of reviews, and augment readability of the code. As stated below, we have regular discussions about these rules and make them evolve when needed.
- ESLint // Prettier // PostCSS
At the root of the monorepo are configuration files for eslint, postcss, and prettier with our rules for code style. These apply to every line of code in the repo and allow us to keep a consistent coding style between different applications. This helps developers feel at home on any part of the codebase.
On a side note, prettier and eslint autofix have really helped us obtain a no-fuss working environment, both in terms of code style when reviewing, and ease of use with IDE integrations. The fact that all code is formatted and checked against Botify guidelines every time one of us hits ‘Save’ in our IDE is a big time-saver for the team.
- Babel
Our transpiler rule file is also at the top-level, and apart from some very special apps, all of Botify’s JavaScript code follows the same ECMA spec. We mostly use language features that are LTS, but we have a few babel plugins for proposals in late stages. Our babelrc allows us to make sure all of Botify’s code is written in the same JavaScript, and minimizes errors in production due to developers not having the language tools they need on older stacks. Again, like our styleguide it also allows our code to be more uniform.
Common scripts
Our top-level ./scripts folder looks like this:
├── script/│ ├── build│ ├── deploy│ ├── install│ ├── lint│ ├── release│ ├── start│ ├── stats│ ├── test│ └── utils
All of these scripts pretty much do what their names say, and they work for all applications in the repo. That means that linting, building, testing, deploying, and releasing an application all goes through the same scripts. It also enables anyone in the Botify Product team to make a new release, and deploy it, quickly, simply, and regardless of the application being released.
Bonus (because I know you are curious and wondering): ‘stats’ is a commonized script to run a build with module statistics (webpack — stats) to display a treemap of the weight of the built bundle. It allows us to study our dependency tree, and perform dependency optimization to reduce bundle size quickly.
- Webpack
Our build script is completely common, and all apps are built through the same script, using webpack. However since the artifacts we build are so different (browser bundles, node bundles, chrome extension, etc..) we’ve built a way to commonize as much as possible about our webpack builds.
We have a top-level webpack.config.js that contains the most common version of our configuration, and it will pull the application’s own webpack.config.js if there is one to make modifications to the build rules before passing the configuration to webpack:
Top-level webpack.config.js:
module.exports = (app, { _ = [] } = {}) => { const appWebpackConfig = require(`./apps/${app}/webpack.config`); const baseConfig = { … }; return appWebpackConfig(baseConfig, { isProduction });};
Which calls appWebpackConfig, a function exported by the child application’s webpack.config.js that can modify rules in the configuration according to its needs.
This allows us to run `yarn build <app>` from the top-level and have a build of the app, using common build configurations while customizing the build for the needs of the app.
All builds, of course, are produced in the application’s ./dist directory.
- CI/CD/Workflow integration scripts
Linting, building, testing, and deploying is done on CI. Again, we’ve created top-level scripts for each of these actions to be performed for each application.
When running on CI, we’ve built a simple function to determine which application has been changed and which application needs to be tested on CI. When a CI build starts, Travis performs a git diff to find which apps have modified files in the branch. It then checks these modified files against dependencies declared between applications (eg: all application depend on common-js and are rebuilt if the common package is modified) to build a dependency tree. The tree is then flattened and CI only tests applications that need to be tested in the PR. This saves us a lot of build time on Travis.
We use Travis Build Stages to run this in a coherent and modularized manner. We are looking into experimenting with Build Matrices to allow for a fully blown-up CI architecture into a single container for each application.
# What we’ve learned
Create a common code package, use it as much as possible
DRY is a principle we hold to heart Botify. We as a team very strongly believe that the best code is no code at all, and reusing as much as possible between our apps eliminates extra code surface that we have to maintain.
We created two ‘apps’ in the monorepo, common-style, and common-js, that are repositories for all the common code used in other applications. They contain anything from utility functions for string manipulation, to common components and views used between applications.
Important gotcha: we use webpack aliases to alias our common code packages, so that imports to the common packages are also the same everywhere (this makes refactors much easier). In all our applications, code from the common bundle can be imported via “import { …} from ‘common-js/…’”. When refactoring code in common-js, we can easily find and replace imports because they are never relative. This rule is declared in the common webpack configuration and automatically added to all applications.
- Common utilities
As much as possible, we want to reuse small functions and utilities between our different applications. Our common-js/utils folder looks like this:
utils├── async.js├── chart.js├── common.js├── constants.js├── datamodel.js├── dimension.js├── dom.js├── filter.js├── filterBlock.js├── layout.js├── moment.js├── predicates.js├── propTypes.js├── query.js├── react.js├── reporting.js├── routing.js├── searchableTree.js├── segments.js└── trackUser.js
A big set of utilities, very various in what they do, some purely technical (eg: dom, react), and some more functional/business (eg: filter, datamodel, segments). They are used all over Botify applications in the repo and they help us be able to make evolutions to large systems quickly.
We also have some utilities that are just wrappers around implementations that differ in different applications. For example, we have a legacy application using BackboneJS, and everything we’ve written in the past two years uses React-Router, so we have a utility to abstract routing away into these two specific implementations. Our routing utility can route, no matter which application we’re in, and the stack details are left unknown to the consumer of the module — because they’re not important, all they want to do is route.
- Common views
Our UI team internally has built a UI Kit, a set of components common to our UIs that can be used when composing views.
Our common package contains React components that are to be reused throughout our applications. This is very important and allows for a more unified design between each component of the repo.
For example, the <Button /> we use in our Botify Extension is the exact same component as the one used in our login page.
This allows us to update core UI in one place, and have the changes apply to all of our apps.
As a bonus, it’s allowed us to build a Storybook app to collaborate with the UI team on evolutions to the components. Each common component has a demo in the Botify Storybook, and the UI team has access to an internally-hosted version. This gives us a quick feedback loop when working on the core components that make the Botify UI.
- Common redux code
We use Redux and Redux-Sage extensively, and we’ve managed to commonize a lot of business logic, data-wrangling, and charting code so we can reuse it in different applications.
A lot of core concepts of any group of products within a suite are common: Users, Groups, Authentication, Authorization, etc. This logic is usually handled in Redux, and we have reducers and a store structure that is reused between different applications.
Our stores for each app is usually structured in the same way, and reducers work across the board the same way for one application or another.
We also use Redux-Saga for the initialization of all of our applications. Initializing a Botify application usually means fetching a lot of metadata about the Botify project being requested, and the different components it has (Crawl, Logs, Keywords, Analytics, etc…). One of our products’ strength is Botify’s ability to cross data between multiple sources (crawls, log data, analytics, etc…). Our common sagas allow us to fetch data and wrangle it in the same way regardless of the product, and therefore allows us to create vizualizations that cross these datasets more easily.
Our common sagas fill our stores in a standardized way, and our common reducers can easily work on any store under the standard.
Always bundle the react version from the app
Nobody has said this better than @danabramov (here):
Two Reacts Won’t Be Friends
A common problem in React monorepos is inadvertently loading two copies of React. Your common code has a `react` dependency, your libraries have `react` dependencies, and your application has a `react` dependency, so it’s easy for Webpack, if it is not well configured, to load the wrong copy of `react`.
We solved them using our common Webpack configuration, which forces the bundler to look for React in the application’s dependencies first and foremost.
Unify your react versions
Unifying your React version between packages is important for a simple reason: components that are reused from application to application, which reside in the common bundle, need to work the same way in all applications. It hasn’t happened in awhile as React has made less and less breaking changes, but quirks between React versions can still happen, especially with a <16/>16 mix with React Fiber. Unifying our React versions across the board eliminates these kinds of headaches before getting them.
On a side note: we’ve used Facebook’s codemods in the past to update legacy code, and it’s worked very well for us so far.
Do not use the factory pattern
Anyone working with this kind of code split will be tempted to use the Factory pattern — we were! And we’ve found that it was a huge mistake.
Here’s an example that illustrates why:
You have a formatting, function. It formats fields according to their type (integer, float, string, etc…). For 99% of our fields, this code will be common, so it resides in the common package:
const formatField = (field) => ...However, in some applications, we are going to want to override this formatter for the special needs of the application. Let’s say we augment this behaviour with our own formatter in an application:
import { formatField } from ‘common’;const applicationFormatField = (field) => { if (…) { return applicationSpecificFormat; } return formatField(field);}
Now any consumer that wants to format needs to import the correct formatter from its own app.
Say we make a formatting component afterwards:
const formattingComponent = (value) => <div> {formatFunction(value)}</div>Now if we want to make this component common, we end up with:
const formattingComponent = (formatFunction) => (value) =>
<div> {formatFunction(value)}</div>From our experience, this pattern scales very badly when there are many things to inject. With time, we ended up needing to provide a lot of different app-specific modules (formatters, fetchers, cachers, transformers) for common components. Factories quickly become very impractical in this case.
The solution here is to use some form of dependency injection, like React’s ‘context’ API (I would not recommend, for other reasons), or Classes (difficult to use with components since there is no multiple inheritance in JavaScript).
We opted for a simple homebrewed dependency injector:
import { mapValues } from 'lodash';
// Plain object that will store all provided dependency
const _dependencies = {};export function provide(dependencyName, dependency) {
_dependencies[dependencyName] = dependency;
}
export function inject(dependencies) {
return (wrapped) => {
return (...args) => {
const resolvedDependencies = mapValues(dependencies, (dependency) => {
return _dependencies[dependency];
});
return wrapped(resolvedDependencies)(...args);
};
};
}
At some point, an application does:
provide('formatField', formatField);And consuming components can just request it from the dependency injector:
const getFormatField = inject({ format: 'formatField' })(
({ format }) => format)
);
const formattingComponent = (value) =>
<div> {getFormatField()(value)} </div>This removes the need to change the surface API of the common component when making these components common, and it helps make refactors simpler.
This short piece of code has its limits and we use it sparingly, but it allows us to inject application-specific dependencies into any function (and therefore any React component).
Have frequent discussions about amending conventions, and do it
As mentioned in our previous post, we at Botify think it is very important to have frequent discussions about our rules and our styleguide, and frequently amend them. We hold monthly meetings where any developer can submit these kind of amendments and we can discuss them.
In the past two years this method has made sure we’ve kept up with changes in the JavaScript language, and today we are using the latest LTS JavaScript features in our codebase.
There are still a lot of things that we’ve discussed internally but not had a chance to implement or try out, including:
- Using yarn workspaces to segment our application workspaces
- Using lerna or some other tooling for js monorepos
- Using Travis build matrices to build applications concurrently on CI
How are you running your repo? There might be limits to our approach that we have not run into yet. Feel free to chime in in the comments!
Interested in joining us? We’re hiring! Don’ t hesitate to shoot me an email if there are no open positions that match your skills, we are always on the lookout for passionate people.
