Simplify the development life cycle with Nx, a monorepo tool
We’d like to share our recent experience with Nx. Nx is a Typescript based monorepo tool, primarily supporting Node.js based frontend, backend apps, and libraries. The topics we are covering today:
- Why monorepo?
- Nx: A monorepo tool
- Create Nx workspace for a project
- Build, lint, test, serve and run
- Single package.json policy
- Affected apps by dep-graph and git diff
- Computation caching and Nx Cloud
- API Gateway + Lambda ready package
- Gitlab CI Pipeline for a monorepo
- Lessons learned
Why monorepo?
Lapis (our company) builds and manages custom end-to-end solutions for our clients and we use AWS and various cloud-based services.
Our architecture is a more modular monolithic approach than fully decoupled microservices, as our solutions are often used for a small number of focused business users (100s ~ 10,000s) and scalability in both systems and team size is not our main concern.
Our projects
- 10 ~ 20 active development projects per year
- Intellectual property (IP) of the project artifacts belong to clients and we often need to hand over code repositories
- Systems are typically in operations for 5 ~ 20 years and evolve over a period of time
Typical Application Components
- Angular webapps (Role-based administrative business workflow)
- Ionic Mobile apps (In the field offline-capable, native or PWA)
- Express/NestJS/Kotlin APIs for webapps (BFF)
- Common APIs and internal libraries
- System integrations (Authentication and service integrations)
- Database(s) including DDLs, SQLs and scripts for migrate/backup/restore
- AWS resource configuration (CloudFormation templates), credentials, and documentation
Manyrepo, Angular workspace, and Meta
We’ve been using manyrepo approach for a project like above, by creating a repository for each component. The project structure looks like below:
Manyrepo cons
- Maintain configurations and scripts for each archetype such as building, testing, and deployment
- Coordinating a release process can become complicated as the knowledge of what components must be released, relies on the development team
- Setting up an entire project locally might take a long time as each component has its own README.md and has a variation (No standard process)
Nx: A mono repo tool
Nx is a Typescript based mono repo tool, built on top of Angular DevKit (CLI and Schematics) and provides a workspace, CLI, a cloud-based computation caching, and a great language level IDE support. Nx was created by Angular team at Google and then the core members decided to start a company (Nrwl) providing Nx toolchain, consulting and education.
Why Nx?
We looked at several tools available and found Nx supported most of our typical application components, Angular webapps, Ionic and Node.js based back-end (Express or Nest).
Nx website says ‘Develop like Google’. Apparently, Google manages their entire codebase in 1 repository and this copy sounds like encouraging every other company should follow.
One monorepo to rule them all vs One monorepo per project
We look at a life-cycle of each project and how it should evolve over a period of time. The following are a few reasons we’ve decided to take 1 monorepo per project approach:
- We want to use the latest technologies for a new project and not constrained by existing modules and breaking changes before starting a new project
- Some projects (systems) do not simply evolve. They are Fit for Purpose and users are happy for a long time.
- Typescript, frameworks and libraries continuously evolve and you often get breaking changes (not backward compatible), which might require a substantial effort/cost to re-write
- Having said that, it’d be ideal if you can afford to upgrade entire codebase for every dependency update so your team only need to manage 1 set of dependencies/platform. (It must be a dream of every developer who’s worked a little longer than 1 year in the Javascript development space!)
Create Nx workspace for a project
create-nx-workspace generates an Nx workspace, which is a directory containing your apps, libs, tests, configuration etc:
npx create-nx-workspace@latest
npx: installed 193 in 30.053s
? Workspace name (e.g., org name)
blueberry
? What to create in the new workspace angular-nest
[a workspace with a full stack application (Angular + Nest)]
? Application name
blueberry
? Default stylesheet format
SASS(.scss) [http://sass-lang.com]
? Use the free tier of the distributed cache provided by Nx Cloud? Yes [Faster command execution, faster CI. Learn more at https://nx.app]It generates a set of configuration files and apps, libs and tools subdirectories containing application artifacts. The Angular and NestJS apps are ready to serve locally.
Schematics: Generator
Nx leverages Schematics, a tool from Angular DevKit, to generate boilerplate code for artifacts (apps, module, service, components etc.). The create-nx-workspace is essentially running a set of schematics within the workspace:
# Generate Angular app, called blueberry
nx g @nrwl/angular:app blueberry# Generate Nest app, called api
nx g @nrwl/nest:app api --frontendProject=blueberry# Generate a Node module, shared by Angular and Nest
nx g @nrwl/node:library api-interface
Build, lint, test, serve and run
Nx uses git to determine what artifacts are affected by looking at git diff. Our typical development loop looks like the following using Nx tasks:
# Create a feature branch
git checkout -b feature/hello-world# Generate a set of components using nx generate + schematics
nx g <schematics># Implement, test, serve (end 2 end test via browser)
nx test api # test api
nx serve api # run api locally# Commit, push (lint, test) and create a MR to develop branch
git commit and push...# Run a CI pipeline
nx affected:build
nx affected:test
nx affected:deploy
You can create custom tasks using a pre-defined builder or create your own custom builder.
This is a powerful concept as you can define any arbitrary tasks for your apps and run via Nx. For instance, creating a deploy task which deploys an API to AWS Lambda. There are many ‘builders’ developed by Nx community and we’ll be using it later to build a lambda ready zip file containing a NestJS app.
Single package.json policy
Nx manages all of your npm dependencies in a single root package.json. This decision is driven by a single version policy of Google and you can read it in detail in this GitHub issue and this article.
Dependency hell?
Our first impression was this could become an issue in a long term. You wanted to upgrade one of the frameworks, which used a library that was also used in another framework and upgrade of one introduce other breaking changes.
Fortunately, npm had already solved this, called peer dependency. You could learn it here.
Affected apps by dep-graph and git diff
Another powerful command is called dep-graph, which shows the dependency graph of a workspace so you can visualise what libraries are used for an app or what apps are dependant on a library.
Nx uses a combination of projects andimplicitDependencies in nx.json to identify dependencies and used for a companion CLI command called affected.
The goal of affected command is to only build, lint, test and deploy apps affected by changes in the repository. Nx simply uses git diff to identify the changes, find the affected apps by looking at the dep-graph.
We feel this affected feature alone worth using Nx as it is hard to track the changes and re-build only affected in a manyrepo strategy.
Computation caching
Nx supports a powerful concept called computation caching. It is a form of caching to avoid re-build, re-lint, re-test, re-deploy if it has been done once.
Nx documentation says they need to know two things to avoid re-computing:
1. We need to store the results of the computation. 2. We need to know when we are about to compute something we already computed before. There are three things that tell Nx if something has been computed before:
Execute once output anywhere
This concept is similar to Build once run anywhere, often mentioned in the Docker community. Nx computation cache is focusing on minimising the development loop for a monorepo environment, in which the codebase can be evolved significantly in size and you will want to save every second that can affect on the team’s productivity.
Nx Cloud
Nx computation cache naturally happens in a local environment and you can feel the difference every time you run build, lint or test. Nx takes a step forward and try to save computation across the development machines and CI environment, by caching the outputs of your computations on Nx Cloud.
The following screenshot shows the Nx cloud console, providing a summary of times spent on each build. This is a snapshot of a project which has been running for 4 sprints (10 days per sprint) with 2 ~3 full-time developers.
Unfortunately, we’re not saving too much time and the tool does not currently provide any more details, such as how many machines are used, their platforms, the reason they couldn’t save time etc. however it’s a great metric to capture/visualise across our development process.
API Gateway + Lambda ready package
We often use AWS API Gateway and Lambda (serverless) as it is cost-effective and the development-deployment life cycle is simple and easy to maintain.
This repository, AWS Serverless Express for Fastify provides a boilerplate code for Fastify and the author is kindly providing an example for NestJS.
The example provides 3 essential configurations to successfully deploy Node.js based backend API to Lambda:
- CloudFormation template for creating API Gateway, IAM Role and Lambda
- Open API (aka Swagger) specification for API Gateway Lambda proxy integration
- Lambda entry point for passing HTTP Request to NestJS (Fastify)
Build API with Lambda entry point
Nx builds NestJS using @nrwl/node:build, which produces main.js containing all of your implementations, but it does not include dependencies (node_modules). You can run it by node dist/apps/api/main from the Nx root directory.
The issue is the root node_modules contains modules for the whole project, including Nx, TypeScript and Angular, which are not required by NestJS.
There are several discussions in the Nx community for how to solve this issue. We recommend reading ‘package.json per app’, which tells you why Nx does not support multiple package.json and how people solve this.
We’ve used Apployees-Nx, @apployees-nx/node searches your code and dynamically generate package.json, which only contains the NestJS dependencies.
The following scripts (in package.json) builds NestJS; install node_modules in dist dir; zip resources; so you can deploy it to Lambda:
GitLab CI and Nx
The CI pipeline can leverage Nx affected capability so the CI only build, lint, test and deploy affected artifacts. You use base and head options to find what affected and Nx finds them by git diff. We create these scripts in package.json so we can call them in CI:
The challenge
In manyrepo, you can make each pipeline simple and straight forward and use triggers to start another pipeline easily and flexibly.
In monorepo, it is harder as each component might require a specific platform or environment and the pipeline can become very complicated quickly.
Lessons learned
- Less configuration/boilerplate: Monorepo reduces a lot of redundant configuration and scripts found in manyrepo strategy as every configuration/scripts are available across the apps and you can easily parameterise them as the code structure is predictable
- Promote modular architecture: Thanks to a generator, you can effortlessly make a shared library and it is instantly available for all apps
- Shared components can be naturally evolved: You can first create a component in an app and easily promote to a library, which includes models, interfaces, validations and DSL specific business logic
- Nx works well for any technology: You can add any artifacts into Nx workspace including database migration scripts, Python-based Lambda or even Kotlin based API (a legacy Maven project) and leverage Nx tooling such as affected, computation cache and standardised build, lint, test deploy cycle
- Better DSL for CI Pipeline: We feel CI tool needs to be refined to fully utilise a tool like Nx, we believe GitLab CI is working hard to better support Monorepo pipeline and look forward to seeing improvements
Thanks for reading!
References
- Cons of monorepo: https://medium.com/@mattklein123/monorepos-please-dont-e9a279be011b
- Monorepo for a service project: https://itnext.io/building-scalable-multi-platform-projects-with-angular-and-nx-a13b40885e54
