Journey of a Frontend Monorepo: Here’s What I Learned

The recipe is monorepo

Photo by Kara Eads on Unsplash

This article will talk about what I learned while restructuring our frontend architecture at Jotform.

We have agile cross-functional teams, and we’re following the Trunk-Based Development approach, which means we are directly pushing updates to our main branch. Our main branch is always production ready and guarded by automated tests. In our scale, our production environment gets updated almost 300 times in a regular day.

We used to have a polyrepo structure, which created a different repo for every application and a few more for shared libraries. Every repo had its own configuration. It was good for separately managing if a repo has an owner. Our company was growing fast, and we didn’t manage repo ownerships well. Including more developers in the team resulted in breaking things. Here were the first problems we faced:

• Inconsistent tooling: We had multiple linter, webpack, testing, and GitHub Action configurations. We did similar things in different repos and had multiple build pipelines.
• Difference in development environment: Every team managed the development dependencies of their project with their own effort.
• Hard-to-share code: We aimed to share code across multiple teams and projects, so we created common components and utilities inside another shared repository. Usually, we linked modules from webpack configs during development and publishing the package to a private registry to use in another application. The process was really slow and painful, so it resulted in a lot of code duplication.
• Unable to manage versions: We were creating private npm packages for code sharing. Once a shared library got an update, we updated every application individually. Our dependent application count was increasing very fast — we were updating 40+ different repositories for a single package update. Unfortunately, we often forgot to update or chose not to update some applications. This resulted in our users seeing different UI for the same component on different applications. This was actually breaking our trunk-based development approach. If we forgot a low-used app, the app would use old packages for weeks.

Polyrepo build process for shared libraries

• Hard-to-see-affected applications: Once a developer made an update to a component library, it was hard to see all affected applications company-wide, which usually resulted in breaking features.
• Hard mode onboarding: Newcomers struggled to find related components in these multiple codebases.

With a lot of pain, we started looking for a recipe.

The Recipe is Monorepo

We started our journey with Yarn Workspaces; it offered good speed in development, and we combined it with Lerna in a fresh repository.
As a first step, we created our core config packages( Eslint, TypeScript, Webpack, etc). A few weeks later, we switched to Turborepo.

Then we started a long-running journey — moving our packages to our monorepo took exactly a year. We started at the beginning of 2022 and finished at the end of 2022.

Our monorepo structure and tooling evolved a lot during the year. We dropped almost all our choices due to performance concerns and our experiences.

Here is the summary of our latest tooling:

• PNPM as the package manager. It is super fast and efficient. It also has great workspace management tooling. You can use your packages from the latest version using the workspace protocol in package.json.
• NX as the monorepo orchestrator. NX is great so far; it offers the following key features for us:
  Computational remote caching: If a commit doesn’t affect a library, NX helps you use previous builds of this library which speeds up your build times. We have almost 250 packages in our monorepo, and we only build affected packages by the commit in seconds.
  Dependency graph: This is very helpful for developers; they can see which packages will be affected by their updates.
• Syncpack helps us use the same dependency versions across all applications and libraries. This is very helpful for reducing bundle sizes and improving the developer experience.
• Danger JS helps us request reviews for critical updates. Like, if an update affects multiple packages, someone created a new package or someone added a new dependency.
• Renovate Bot helps us manage our dependencies up-to-date and secure.
• Internal CLI: We created an internal CLI to let developers easily create new applications or libraries.

Currently we have almost 250 packages, and we placed our packages in a similar structure described in the following image:

Sample monorepo package structure

What did I learn during this process?

Pros

A Monorepo can speed up your development, and it can improve developer experience if it is well structured, and most importantly, it is powered by good tooling.

• Great for standardization: We don’t have multiple configs or build processes anymore. Developers use the same code and commit standards in company-wide. Monorepo approach reduced our technical debts. Now, we’re sure every application is using the same configurations. For example, if we update our browser support, we know every part of the company gets the same browser support.
• Code sharing in seconds: It is easy to create new shared libraries and share code using pnpm workspaces feature and some internal boilerplates.
• Common development environment: It is easy to create common commands and standardize development environment to single command.
• Better onboarding: Newcomers can adapt easily with a single and well-structured codebase.
• Easier refactoring: You can send your commits to multiple projects at once, so you don’t need to wait for another PR to be merged on another repository.
• Reduced total build times: NX like orchestrator tools helps you to reduce your build times using computational caching algorithms. It builds and caches build outputs both locally and remotely. Your total build time lessens, especially for shared libraries.

Build process on commit to shared library

• Dependency awareness: It is effortless to see how your application and libraries are connected to each other. Developers are aware of which applications or libraries will be affected by their commits. We’re also giving this information via a custom GitHub Action for improving awareness.

Cons

Monorepos also have some bad parts. It is important to keep your monorepo away from becoming a monolith.

• Hard-to-migrate existing repos: One of the hardest parts is to import existing repos to new monorepo. The existing repo needs to be ready with your new monorepo requirements. Before the import phase, you should update all configs and dependencies to your new monorepo-compatible versions which is a pain and sometimes take weeks.
  A tip: You can import your existing repo and preserve the existing commit history (if it is important to you) using the lerna import tool.
  The following article is helpful if you want to learn more about the importing process: 4 things we’ve learned while migrating a project inside a monorepo
• Working times: It becomes harder to make critical changes if you are changing an actively working system or if the source repo is actively in development. You probably need to create most of your work on weekends or general off-times if the source repo is actively in development. You don’t want to interrupt business :).
• Managing build queue: Our main branch gets 300 commits in a regular day. We’re following trunk-based development, but if we allow all developers to directly push to the master branch in monorepo, they may race to push their commits, creating huge build queues. If someone fails on the trunk, development stops. We have to use short-lived feature branches to keep our trunk-based development working.
  Basically, we make the build and test phases in pull requests and use the main CI job only to sync to the servers. This is needed to scale our system.
  We’re using GitHub’s auto-merge option and a custom-built solution to scale project builds. In this approach, if a PR fails, we don’t fail. The main job still becomes clear.

A simple timeline of parallel builds

• Hard-to-debug caching: We’re aiming for our build times to be under two minutes. We had some issues with NX computational caching — cahes were becoming invalid even if the related module didn’t get an update. This resulted in increased build times, and we had difficulties finding the root cause.
• Less freedom: People like new technologies, and it is easier to try new things on a single repository. If you want your monorepo to keep up with standards, it becomes harder to apply new things. For example, we wanted to use the same dependency versions across our codebase. If someone wanted to upgrade a package (ex: React), they needed to update the whole codebase. Of course, you can also allow this in monorepo, but it breaks the theory.

Conclusion

Overall, our frontend monorepo approach improved our developer experience, but you should know your pros and cons before ddeciding.

It is probably better but will take longer than you think.

I believe some steps in this article, like orchestation and build tooling, deserve further explanations. I look forward to writing in-depth articles on these topics soon!

Want to Connect?

I'm Berkay Aydin, engineering director and frontend architect at Jotform.

Feel free to send me a message or find me on twitter/sbayd or LinkedIn.