Monorepo Insights: Nx, Turborepo, and PNPM (2/4)
Exploring the strengths and weaknesses of today’s top monorepo solutions
Explore the complete series through the links below:
- Monorepo Insights: Nx, Turborepo, and PNPM (1/4)
- Monorepo Insights: Nx, Turborepo, and PNPM (2/4) (we are here)
- Monorepo Insights: Nx, Turborepo, and PNPM (3/4)
- Monorepo Insights: Nx, Turborepo, and PNPM (4/4)
Introduction
This article is part of a series where we compare the features, performance, and suitability of Nx, PNPM, and Turborepo for our projects.
Having explored the fundamentals of monorepo management and build systems, this article delves into NX, a powerful tool in this space. ✨
As a reminder, our goal is to select a champion who will streamline our development workflow and improve our codebase management.
May the best monorepo shine! Here’s the challenge we’ll conquer together:
· NX under the microscope
∘ NX running locally (without Daemon)
∘ NX running locally (with Daemon)
· Nx Cloud
∘ Nx Replay (Remote Caching)
∘ Nx Agents (Distributed Task Execution)
∘ Benefits of Distributed Caching and Task Execution
· Hands-on NX
∘ Nx Locally in Action
∘ Distributed Caching in Action
· Technical verdict
∘ Our insights
∘ Real-World Insights: Nx in the Wild
∘ Key takeaways: A Powerhouse with Nuances
· Conclusion
Curious about what’s next? Come along and let’s discover it together! 🚀 🌟
NX under the microscope
The complete picture of NX’s features is as follows:
And the Nx workflow’s high-level overview is as follows:
+-------------------+ +------------------+ +----------------+
| Workspace Config | -------> | Project Graph | ----------->| Task Graph |
| (workspace.json, | | (DAG: projects | | (DAG: tasks) |
| project.json) | | & deps) | | |
+-------------------+ +------------------+ +----------------+
|
|
v
+------------+
| Task |
| Execution |
| & Caching |
+------------+In this section, we’ll analyze the key components to understand how they work, allowing us to assess their strengths and limitations.
NX running locally (without Daemon)
NX is a sophisticated toolkit designed to streamline monorepo development and management.
It achieves this through intelligent project graph analysis, optimized task execution, efficient caching, and a flexible plugin system. Let’s go deeper!
🔳 The first observation is that when installing Nx, the following output is typically seen in the console:
nx % pnpm i -D nx
Packages: +109
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Progress: resolved 118, reused 0, downloaded 109, added 109, done
node_modules/.pnpm/nx@19.3.2/node_modules/nx: Running postinstall script, done in 526ms
devDependencies:
+ nx 19.3.2
Done in 9.1s🔳 A crucial post-installation script, packages/nx/bin/post-install.ts, is executed, performing some essential setup works:
// https://github.com/nrwl/nx/blob/master/packages/nx/bin/post-install.ts
...
(async () => {
const start = new Date();
try {
setupWorkspaceContext(workspaceRoot);
if (isMainNxPackage() && fileExists(join(workspaceRoot, 'nx.json'))) {
assertSupportedPlatform();
try {
await daemonClient.stop();
} catch (e) {}
const tasks: Array<Promise<any>> = [
buildProjectGraphAndSourceMapsWithoutDaemon(),
];
...Two key functions within this script are setupWorkspaceContext and buildProjectGraphAndSourceMapsWithoutDaemon.
🔳 setupWorkspaceContext: Bridging TypeScript and Rust
This function acts as a bridge between Nx’s TypeScript code and its underlying Rust implementation. It dynamically loads the compiled Rust module, which contains the core logic for project graph management:
// https://github.com/nrwl/nx/blob/master/packages/nx/src/utils/workspace-context.ts#L9
export function setupWorkspaceContext(workspaceRoot: string) {
const { WorkspaceContext } =
require('../native') as typeof import('../native');
performance.mark('workspace-context');
// https://github.com/nrwl/nx/blob/master/packages/nx/src/native/workspace/context.rs#L163
workspaceContext = new WorkspaceContext(
workspaceRoot,
cacheDirectoryForWorkspace(workspaceRoot)
);
performance.mark('workspace-context:end');
performance.measure(
'workspace context init',
'workspace-context',
'workspace-context:end'
);
}The key action here is the creation of a WorkspaceContext instance in the Rust layer. This instance becomes the central interface for interacting with the workspace, including file operations and project graph construction:
// https://github.com/nrwl/nx/blob/master/packages/nx/src/native/workspace/context.rs#L163
#[napi]
impl WorkspaceContext {
#[napi(constructor)]
pub fn new(workspace_root: String, cache_dir: String) -> Self {
enable_logger();
trace!(?workspace_root);
let workspace_root_path = PathBuf::from(&workspace_root);
WorkspaceContext {
files_worker: FilesWorker::gather_files(&workspace_root_path, cache_dir),
workspace_root,
workspace_root_path,
}
}🔳 gather_files : Asynchronous File Collection (Rust)
Within the Rust implementation (context.rs), the gather_files function (called from the WorkspaceContext constructor) is responsible for efficiently gathering and hashing files in the workspace:
// https://github.com/nrwl/nx/blob/master/packages/nx/src/native/workspace/context.rs#L36
// https://github.com/nrwl/nx/blob/master/packages/nx/src/native/workspace/context.rs
impl FilesWorker {
fn gather_files(workspace_root: &Path, cache_dir: String) -> Self {
if !workspace_root.exists() {
warn!(
"workspace root does not exist: {}",
workspace_root.display()
);
return FilesWorker(None);
}
// ... (logging and setup)
let archived_files = read_files_archive(&cache_dir);
// ... (thread synchronization setup)
thread::spawn(move || {
// ... (lock acquisition)
let file_hashes = if let Some(archived_files) = archived_files {
selective_files_hash(&workspace_root, archived_files)
} else {
full_files_hash(&workspace_root)
};
// ... (file data processing and sorting)
*workspace_files = files; // Store the file data in the shared vector
// ... (notify main thread and write cache)
cvar.notify_all();
....
write_files_archive(&cache_dir, file_hashes);
});
FilesWorker(Some(files_lock)) // Return the FilesWorker instance
}
}🔵 Key steps include:
✔️ Workspace Root Check: ensures the workspace directory exists.
✔️ Cached File Reading (Optional): reads previously cached file information to speed up incremental updates.
✔️ Background Thread Creation: spawns a separate thread to handle file hashing asynchronously (thread::spawn ).
✔️ File Hashing:
- Calculates hashes for all relevant files (using
selective_files_hashif cached data is available, orfull_files_hashotherwise). - Sorts the file data (path and hash).
✔️ Data Storage: stores the sorted file data in the shared workspace_files vector.
✔️ Notification and Caching: notifies the main thread (cvar.notify_all()) that the data is ready and writes the file data to the cache (write_files_archive).
🔳 buildProjectGraphAndSourceMapsWithoutDaemon: Orchestrating Project Graph Creation
buildProjectGraphAndSourceMapsWithoutDaemon function orchestrates the entire process of building the Nx project graph and associated source maps when the Nx daemon is not used:
// https://github.com/nrwl/nx/blob/master/packages/nx/src/project-graph/build-project-graph.ts#L1
// https://github.com/nrwl/nx/blob/master/packages/nx/src/project-graph/project-graph.ts#L92
...
export async function buildProjectGraphAndSourceMapsWithoutDaemon() {
global.NX_GRAPH_CREATION = true;
const nxJson = readNxJson();
...
let configurationResult: ConfigurationResult;
let projectConfigurationsError: ProjectConfigurationsError;
const [plugins, cleanup] = await loadNxPlugins(nxJson.plugins);
try {
configurationResult = await retrieveProjectConfigurations(
plugins,
workspaceRoot,
nxJson
);
} catch (e) {
if (e instanceof ProjectConfigurationsError) {
projectConfigurationsError = e;
configurationResult = e.partialProjectConfigurationsResult;
} else {
throw e;
}
}
const { projects, externalNodes, sourceMaps, projectRootMap } =
configurationResult;
....
const { allWorkspaceFiles, fileMap, rustReferences } =
await retrieveWorkspaceFiles(workspaceRoot, projectRootMap);
...
const cacheEnabled = process.env.NX_CACHE_PROJECT_GRAPH !== 'false';
...
let projectGraphError: AggregateProjectGraphError;
let projectGraphResult: Awaited<
ReturnType<typeof buildProjectGraphUsingProjectFileMap>
>;
try {
projectGraphResult = await buildProjectGraphUsingProjectFileMap(
projects,
externalNodes,
fileMap,
allWorkspaceFiles,
rustReferences,
cacheEnabled ? readFileMapCache() : null,
plugins,
sourceMaps
);
} catch (e) {
if (isAggregateProjectGraphError(e)) {
projectGraphResult = {
projectGraph: e.partialProjectGraph,
projectFileMapCache: null,
};
projectGraphError = e;
} else {
throw e;
}
} finally {
// When plugins are isolated we don't clean them up during
// a single run of the CLI. They are cleaned up when the CLI
// process exits. Cleaning them here could cause issues if pending
// promises are not resolved.
if (process.env.NX_ISOLATE_PLUGINS !== 'true') {
cleanup();
}
}
...🔵 Key steps include:
✔️ Loading configuration and plugins: reads the nx.json configuration and loads relevant plugins.
const nxJson = readNxJson();
....
// https://github.com/nrwl/nx/blob/master/packages/nx/src/config/nx-json.ts#L459
export function readNxJson(root: string = workspaceRoot): NxJsonConfiguration {
const nxJson = join(root, 'nx.json');
if (existsSync(nxJson)) {
const nxJsonConfiguration = readJsonFile<NxJsonConfiguration>(nxJson);
if (nxJsonConfiguration.extends) {
const extendedNxJsonPath = require.resolve(nxJsonConfiguration.extends, {
paths: [dirname(nxJson)],
});
const baseNxJson = readJsonFile<NxJsonConfiguration>(extendedNxJsonPath);
return {
...baseNxJson,
...nxJsonConfiguration,
};
} else {
return nxJsonConfiguration;
}✔️ Retrieving and parsing project configurations to gather project information:
...
// https://github.com/nrwl/nx/blob/master/packages/nx/src/project-graph/utils/retrieve-workspace-files.ts#L63
export async function retrieveProjectConfigurations(
plugins: LoadedNxPlugin[],
workspaceRoot: string,
nxJson: NxJsonConfiguration
): Promise<ConfigurationResult> {
const globPatterns = configurationGlobs(plugins);
const workspaceFiles = await globWithWorkspaceContext(
workspaceRoot,
globPatterns
);
return createProjectConfigurations(
workspaceRoot,
nxJson,
workspaceFiles,
plugins
);
}
...
// https://github.com/nrwl/nx/blob/master/packages/nx/src/project-graph/utils/retrieve-workspace-files.ts#L82
export async function retrieveProjectConfigurationsWithAngularProjects(
workspaceRoot: string,
nxJson: NxJsonConfiguration
): Promise<ConfigurationResult> {
const pluginsToLoad = nxJson?.plugins ?? [];
if (
shouldMergeAngularProjects(workspaceRoot, true) &&
!pluginsToLoad.some(
(p) =>
p === NX_ANGULAR_JSON_PLUGIN_NAME ||
(typeof p === 'object' && p.plugin === NX_ANGULAR_JSON_PLUGIN_NAME)
)
) {
pluginsToLoad.push(join(__dirname, '../../adapter/angular-json'));
}
...
✔️ Retrieving workspace files to collect information about all relevant files in the workspace:
// https://github.com/nrwl/nx/blob/master/packages/nx/src/project-graph/utils/retrieve-workspace-files.ts#L26
/**
* Walks the workspace directory to create the `projectFileMap`, `ProjectConfigurations` and `allWorkspaceFiles`
* @throws
* @param workspaceRoot
* @param nxJson
*/
export async function retrieveWorkspaceFiles(
workspaceRoot: string,
projectRootMap: Record<string, string>
) {
...
const { projectFileMap, globalFiles, externalReferences } =
await getNxWorkspaceFilesFromContext(workspaceRoot, projectRootMap);
...
return {
allWorkspaceFiles: buildAllWorkspaceFiles(projectFileMap, globalFiles),
fileMap: {
projectFileMap,
nonProjectFiles: globalFiles,
},
rustReferences: externalReferences,
};
}✔️ buildProjectGraphUsingProjectFileMap function constructs the project graph based on the collected information:
export async function buildProjectGraphUsingProjectFileMap(
projectRootMap: Record<string, ProjectConfiguration>,
externalNodes: Record<string, ProjectGraphExternalNode>,
fileMap: FileMap,
allWorkspaceFiles: FileData[],
rustReferences: NxWorkspaceFilesExternals,
fileMapCache: FileMapCache | null,
plugins: LoadedNxPlugin[],
sourceMap: ConfigurationSourceMaps
): Promise<{
projectGraph: ProjectGraph;
projectFileMapCache: FileMapCache;
}> {
...
return {
projectGraph,
projectFileMapCache,
};
}🔳 buildProjectGraphUsingProjectFileMap, ProjectGraphBuilder, and buildProjectGraphUsingContext: these functions collaborate to build the project graph incrementally, add external nodes, normalize project nodes, and apply implicit dependencies.
🔳 The ProjectGraphBuilder class facilitates the process of adding and removing nodes and dependencies to the graph:
export class ProjectGraphBuilder {
// TODO(FrozenPandaz): make this private
readonly graph: ProjectGraph;
...
/**
* Adds a project node to the project graph
*/
addNode(node: ProjectGraphProjectNode): void {
// Check if project with the same name already exists
if (this.graph.nodes[node.name]) {
// Throw if existing project is of a different type
if (this.graph.nodes[node.name].type !== node.type) {
throw new Error(
`Multiple projects are named "${node.name}". One is of type "${
node.type
}" and the other is of type "${
this.graph.nodes[node.name].type
}". Please resolve the conflicting project names.`
);
}
}
this.graph.nodes[node.name] = node;
}
/**
* Removes a node and all of its dependency edges from the graph
*/
removeNode(name: string) {
if (!this.graph.nodes[name] && !this.graph.externalNodes[name]) {
throw new Error(`There is no node named: "${name}"`);
}
...🔳 The ProjectGraph (Directed Acyclic Graph (DAG)) is a critical data structure that enables Nx to perform efficient task scheduling, dependency analysis, and incremental builds.
/**
* A Graph of projects in the workspace and dependencies between them
*/
export interface ProjectGraph {
nodes: Record<string, ProjectGraphProjectNode>;
externalNodes?: Record<string, ProjectGraphExternalNode>;
dependencies: Record<string, ProjectGraphDependency[]>;
version?: string;
}Every time you invoke a target directly, such as `nx test myapp`, or run affected commands, such `nx affected:test`, Nx first needs to generate a project graph in order to figure out how all the different projects and files within your workspace fit together. Naturally, the larger your workspace gets, the more expensive this project graph generation becomes. — https://github.com/nrwl/nx/blob/master/docs/shared/concepts/daemon.md?plain=1
🔳 To correctly model the workspace as a DAG (graph in general), it is important to have clear boundaries that have well-defined cohesive units (modules).
If you partition your code into well-defined cohesive units, even a small organization will end up with a dozen apps and dozens or hundreds of libs. If all of them can depend on each other freely, chaos will ensue, and the workspace will become unmanageable. — https://nx.dev/features/enforce-module-boundaries
🔳 By default, NX ensures this tightness, but it is also possible to reinforce this rule:
To help with that Nx uses code analysis to make sure projects can only depend on each other’s well-defined public API. It also allows you to declaratively impose constraints on how projects can depend on each other. — https://nx.dev/features/enforce-module-boundaries
🔳 Task Orchestration and Execution: the project graph is used to create a task graph (createTaskGraphAndValidateCycles), which represents the tasks to be executed and their dependencies:
...
export async function runCommand(
projectsToRun: ProjectGraphProjectNode[],
projectGraph: ProjectGraph,
{ nxJson }: { nxJson: NxJsonConfiguration },
nxArgs: NxArgs,
overrides: any,
initiatingProject: string | null,
extraTargetDependencies: Record<string, (TargetDependencyConfig | string)[]>,
extraOptions: { excludeTaskDependencies: boolean; loadDotEnvFiles: boolean }
): Promise<NodeJS.Process['exitCode']> {
const status = await handleErrors(
process.env.NX_VERBOSE_LOGGING === 'true',
async () => {
const projectNames = projectsToRun.map((t) => t.name);
const taskGraph = createTaskGraphAndValidateCycles(
projectGraph,
extraTargetDependencies ?? {},
projectNames,
nxArgs,
overrides,
extraOptions
);
const tasks = Object.values(taskGraph.tasks);
const { lifeCycle, renderIsDone } = await getTerminalOutputLifeCycle(
initiatingProject,
projectNames,
tasks,
nxArgs,
nxJson,
overrides
);
const status = await invokeTasksRunner({
tasks,
projectGraph,
taskGraph,
lifeCycle,
nxJson,
nxArgs,
loadDotEnvFiles: extraOptions.loadDotEnvFiles,
initiatingProject,
});
await renderIsDone;
return status;
}
);
return status;
}
...The function createTaskGraphAndValidateCycles is the one responsible of creating the task graph. It receives the projectGraph, the extraTargetDependencies and other parameters as input and returns a task graph after validation.
The task graph is validated to ensure it’s acyclic (no circular dependencies). If cycles are detected, Nx either throws an error or warns the user, depending on configuration settings:
const cycle = findCycle(taskGraph);
if (cycle) {
if (process.env.NX_IGNORE_CYCLES === 'true' || nxArgs.nxIgnoreCycles) {
output.warn({
title: `The task graph has a circular dependency`,
bodyLines: [`${cycle.join(' --> ')}`],
});
makeAcyclic(taskGraph);
} else {
output.error({
title: `Could not execute command because the task graph has a circular dependency`,
bodyLines: [`${cycle.join(' --> ')}`],
});
process.exit(1);
}
}🔳 Nx actually utilizes both DAGs (Directed Acyclic Graphs) and Task Graphs in its architecture, each serving a distinct purpose:
✔️ Project Graph (DAG):
- Represents: The static structure of the workspace, where nodes are projects (applications, libraries, etc.) and edges represent dependencies between them.
- Purpose: Used for dependency analysis, code generation, and determining affected projects when changes are made.
- Example: In an Nx workspace, if the
appproject depends on aui-liblibrary, there would be a directed edge from theappnode to theui-libnode in the project graph.
✔️ Task Graph (DAG):
- Represents: The dynamic workflow of tasks to be executed, where nodes are individual tasks (e.g., build, test, lint) and edges represent dependencies between them.
- Purpose: Used for task scheduling, parallelization, and optimization of the build/test/deploy process.
- Example: Building the
appproject might involve tasks likebuild ui-lib,lint app, andtest app. The task graph would define the order in which these tasks need to be executed, ensuring thatbuild ui-libis completed beforebuild app.
💡 Tasks are not executed in isolation. They are part of a larger task graph that represents the relationships and dependencies between tasks. This graph ensures that tasks are executed in the correct order, respecting dependencies while optimizing for parallel execution.
💡 Nx provides different task runners (e.g., defaultTasksRunner, nxCloudTasksRunnerShell) that are responsible for actually executing the tasks in the task graph. The invokeTasksRunner function, which we saw earlier, is responsible for selecting the appropriate task runner based on configuration and context.
🔳 To find the shortest path, NX uses the Dijkstra algorithm:
// https://github.com/nrwl/nx/blob/master/graph/ui-graph/src/lib/graph.ts#L164
...
case 'notifyGraphTracing':
if (event.start && event.end) {
if (event.algorithm === 'shortest') {
elementsToSendToRender = this.projectTraversalGraph.traceProjects(
event.start,
event.end
);
} else {
elementsToSendToRender =
this.projectTraversalGraph.traceAllProjects(
event.start,
event.end
);
}
}
break;
}
...
// https://github.com/nrwl/nx/blob/master/graph/ui-graph/src/lib/util-cytoscape/project-traversal-graph.ts#L146
traceProjects(start: string, end: string) {
const dijkstra = this.cy
.elements()
.dijkstra({ root: `[id = "${start}"]`, directed: true });
const path = dijkstra.pathTo(this.cy.$(`[id = "${end}"]`));
return path.union(path.ancestors());
}After learning about Nx’s dependency graph and some basic abilities, it’s time to discover another important ability: Daemon. 👻
NX running locally (with Daemon)
🔵 The Nx Daemon is enabled by default when running on the local machine. To turn it off:
- set `useDaemonProcess: false` in the runners options in `nx.json` or
- set the `NX_DAEMON` env variable to `false`.🔵 The Daemon is a long-running background process designed to enhance the performance and responsiveness of Nx commands in several ways:
✔️ Project Graph Cache: it maintains a cached version of the project graph in memory, avoiding the need to rebuild it for every command.
✔️ File Watching: it actively monitors changes to files in the workspace and updates the project graph incrementally when needed.
The Nx Daemon is more efficient at recomputing the project graph because it watches the files in your workspaces and updates the project graph right away (intelligently throttling to ensure minimal recomputation). It also keeps everything in memory, so the response tends to be a lot faster. — https://github.com/nrwl/nx/blob/master/docs/shared/concepts/daemon.md?plain=1
✔️ Parallel Tasks: it can execute multiple tasks concurrently, leveraging multiple cores or even machines (with Nx Cloud) to speed up builds.
✔️ Task Cache: similar to the local cache, the daemon maintains a cache for task results, further improving performance by avoiding re-execution of unchanged tasks.
In order to be most efficient, the Nx Daemon has some built in mechanisms to automatically shut down (including removing all file watchers) when it is not needed. These include: after 3 hours of inactivity (meaning the workspace’s Nx Daemon did not receive any requests or detect any file changes in that time) and when the Nx installation changes. — https://github.com/nrwl/nx/blob/master/docs/shared/concepts/daemon.md?plain=1
🔴 There is one unique Nx Daemon per Nx workspace.
The Nx Daemon is a process which runs in the background on your local machine. There is one unique Nx Daemon per Nx workspace meaning if you have multiple Nx workspaces on your machine active at the same time, the corresponding Nx Daemon instances will operate independently of one another and can be on different versions of Nx. — https://github.com/nrwl/nx/blob/master/docs/shared/concepts/daemon.md?plain=1
I think that at this level we have dissected everything about local operation, what do you think about going to the Nx Cloud now? Here we go! ☁️
Nx Cloud
Nx Cloud supercharges monorepo development by leveraging the power of distributed caching and task execution. By sharing and reusing build artifacts like LEGO bricks, and parallelizing tasks like a team of builders, Nx Cloud significantly reduces build times, conserves resources, and enhances team collaboration, making it easier to manage even the most complex projects.
🔴 NX Cloud is a paid service and here are the different plans.
Nx Replay (Remote Caching)
Nx Replay is a key feature of Nx Cloud that enables distributed caching. It stores the results of tasks in a remote cache accessible to all team members and CI machines:
✔️ Faster CI for Modified PRs: Subsequent commits to a PR can reuse cached results from previous CI runs, reducing unnecessary task reruns.
✔️ Reusing CI Results on Developer Machines: Developers can instantly benefit from the results of CI tasks, speeding up local builds and tests.
Nx Agents (Distributed Task Execution)
Nx Agents is another crucial component that allows distributing tasks across multiple machines to optimize CI execution:
✔️ Task Distribution: The main CI machine sends tasks to agent machines provided by Nx Cloud. Nx Cloud intelligently determines which tasks can run concurrently and distributes them to the agents to minimize idle time.
✔️ Dependency Management: Nx Cloud ensures tasks are executed in the correct order based on their dependencies.
✔️ Result Collection: The results from the agent machines are collected back to the main machine, making it appear as if all tasks were executed there.
✔️ Dynamic Scaling: The number and size of agent machines can be dynamically adjusted based on the size of the PR or specific project requirements
// .nx/workflows/dynamic-changesets.yaml
distribute-on:
small-changeset: 3 linux-medium-js
medium-changeset: 6 linux-medium-js
large-changeset: 10 linux-medium-js
// .github/workflows/main.yaml
...
jobs:
- job: main
displayName: Main Job
...
steps:
- checkout
- run: npx nx-cloud start-ci-run --distribute-on=".nx/workflows/dynamic-changesets.yaml" --stop-agents-after="e2e-ci"
- ...To determine the size of the PR, Nx Cloud calculates the relationship between the number of affected projects and the total number of projects in the workspace. It then assigns it to one of the three categories: small, medium, or large.
Now PRs that affect a small percentage of the repo will run on 3 agent, mid-size PRs will use 6 agents and large PRs will use 10 agents. This feature helps save costs on the smaller PRs while maintaining the high performance necessary for large PRs.
Benefits of Distributed Caching and Task Execution
✔️ Faster CI Pipelines: Significantly reduces CI run times, especially for large projects.
✔️ Improved Developer Experience: Faster builds and tests on developer machines.
✔️ Simplified Setup: Minimal configuration required for distributing tasks.
✔️ Scalability: Dynamically scales resources to meet the needs of the project.
💡 Nx Cloud uses sophisticated algorithms for task distribution, taking into account project dependencies, task execution times (estimated or historical), and available agent resources. It aims to minimize the total execution time and balance the load across agents. The specific algorithms can vary, but they often involve heuristics and optimization techniques.
It’s time for us to put our knowledge to the test and start practicing! 🚧
Hands-on NX
Nx Locally in Action
🔳 Installation:
✔️ The Nx CLI can be installed globally using npm install -g nx. This allows nx commands to be used directly.
✔️ Alternatively, Nx commands can be run without global installation by using npx nx.
🔳 Creating a Workspace:
✔️ A new Nx workspace with a specific preset (e.g., React, Angular, Node) can be created using npx create-nx-workspace@latest.
✔️ Nx capabilities can be added to an existing project by running npx nx init.
🔳 Generating projects and libraries: projects and libraries with preset configurations can be created using generators like nx g @nx/react:app my-new-app .
🔳 Running Tasks:
✔️ Nx tasks can be created from existing package.json scripts, inferred from tooling configuration files, or defined in a project.json file. Nx will combine all three sources to identify the tasks for a specific project.
// https://nx.dev/reference/project-configuration
...
"targets": {
"test": {
"inputs": ["default", "^production"],
"outputs": [],
"dependsOn": ["build"],
"executor": "@nx/jest:jest",
"options": {}
},
...✔️ To run a task, Nx uses the following syntax:
✔️ It’s also possible to use run-many command to run a task for multiple projects:
npx nx run-many -t build lint test✔️ The nx affected command can be used to intelligently run tasks only on the projects affected by the code changes:
npx nx affected -t test🔳 Leveraging the Project Graph:
✔️ The dependency graph of the workspace can be visualized using nx graph:
✔️ Available plugins and their capabilities can be viewed using nx list.
🔳 Popular Nx Plugins: Nx offers a wide range of official and community plugins to extend its functionality:
✔️ A complete list of official Nx plugins can be found at: https://nx.dev/plugin-registry.
✔️ You can find many recipes for using NX here: https://github.com/nrwl/nx-recipes/tree/main.
Let’s proceed with configuring distributed versions. ☁️
Distributed Caching in Action
Once connected, Nx will automatically start using remote caching. When a command like nx build my-app is run, the cloud cache will first be checked by Nx. If the results are found, they will be downloaded, compared and may be used, skipping the local build.
🔳 To enable distributed task execution in the CI pipeline:
✔️ The nx-cloud start-ci-run command should be added to the CI configuration before running any Nx commands.
✔️ The number and type of agents to use (e.g., --distribute-on="4 linux-medium") will need to be specified.
✔️ Regular Nx commands (e.g., nx affected --target=build) can then be proceeded with. These tasks will be automatically distributed across the specified agents by Nx Cloud.
# https://nx.dev/ci/recipes/set-up/monorepo-ci-gitlab
# ... (other CI steps)
- name: Start CI run
run: npx nx-cloud start-ci-run --distribute-on="4 linux-medium"
- name: Run affected tests
run: nx affected --target=test --parallel
# ... (rest of your CI steps)It’s time for us to express our opinion on everything we’ve seen and studied.📌
Technical verdict
Our insights
🔵 Nx Strengths: Empowering Large-Scale Development
✔️ Efficiency Boost: Nx significantly improves developer productivity by streamlining tasks, optimizing build processes, and enabling parallel execution.
✔️ Robust Architecture: The use of DAGs and topological sorting ensures correct task ordering and prevents circular dependencies.
✔️ Rich Ecosystem: The wide range of plugins and generators simplifies development and integration with various tools and frameworks.
✔️ Excellent Documentation: The comprehensive documentation makes learning and using Nx easier, even for newcomers.
✔️ Nx Cloud Advantage: Distributed caching and task execution (with Nx Cloud) drastically improve build times and resource utilization, particularly for large projects.
✔️ Integration with Existing Tools: Nx can be integrated with existing build tools and frameworks, resulting in a smooth transition.
🔴 Nx Challenges: Considerations for Adoption
✔️ Learning Curve: Mastering Nx’s concepts and configuration options can be challenging for those new to the tool.
✔️ Black Box Effect: The code generation feature can be a double-edged sword, making it harder for less experienced developers to understand the inner workings of their projects.
✔️ Limited Introspection: Apart from the graph and affected mechanism, Nx offers less in-depth visibility into the build process, potentially hindering debugging.
✔️ Opinionated Structure: Nx’s conventions on project structure and configuration might conflict with established team practices or preferences.
✔️ Overkill for Smaller Projects: Nx’s full range of features might be excessive for smaller projects or teams, where simpler tools might suffice.
✔️ The uses of two types of graphs: the Project Graph (a DAG representing project dependencies) and the Task Graph (also a DAG representing task dependencies).
To gain a broader perspective on Nx’s effectiveness, let’s explore how it’s being used in real-world projects and gather insights from the community since they are also NX users.
Real-World Insights: Nx in the Wild
To gain a balanced perspective on Nx, we’ve dived into both the vibrant online community and the Nx GitHub issue tracker. Here’s what we’ve learned:
1️⃣ Community Feedback (Reddit):
🔳 Links to Reddit Threads:
🔸 “AskJS: Experiences with Nx for Javascript monorepo”
🔸 “What’s your biggest gripes with NRWL NX?”
🔸 “NX vs Turborepo? Concerned about betting on either”
🔳 Summary of feedback:
➕ Nx shines in large-scale, enterprise-level monorepos where its intelligent task orchestration, dependency management, and code generation streamline development and boost productivity. Angular developers particularly appreciate Nx’s seamless integration and automation capabilities.
➖ However, users also point out potential drawbacks. Nx’s opinionated nature and learning curve can be hurdles, especially for those seeking flexibility or a simpler learning experience. Some users find Nx overkill for smaller projects or express concerns about its Git management and commercial backing.
2️⃣ Technical Issues (Nx’s GitHub):
Analyzing Nx’s GitHub issue tracker reveals recurring challenges:
🔻 Issue #26798: (scalability limitations) Nx can struggle with rebuilding project graphs in very large repositories (250+ libraries).
🔻 Issue #26778 and Issue #26771: (compatibility issues) Some users report difficulties using custom libraries in end-to-end tests or building Next.js apps with the Nx CLI.
🔻 Issue #26783: (file handling and configuration) Issues with project renaming/moving and complex configuration management have also been raised.
These insights highlight potential friction points in adopting and using Nx, emphasizing the importance of considering the project’s specific needs and constraints.
Key takeaways: A Powerhouse with Nuances
🔵 Strengths:
✔️ Excels in large, complex monorepos.
✔️ Ideal for enterprise environments.
✔️ Intelligent task orchestration & build optimization.
✔️ Promotes code sharing across projects.
✔️ Rich feature set for scalability and productivity.
🔴 Potential Drawbacks:
✔️ Opinionated structure might limit flexibility.
✔️ Learning curve for new users.
✔️ Feature-rich, potentially overkill for smaller projects.
⚪ Consider Alternatives If:
✔️ The project is small or simple.
✔️ Flexibility is a top priority.
✔️ Cost-effectiveness is crucial.
Our Nx exploration ends here, but our monorepo journey continues! Stay tuned for in-depth analyses of Turborepo and pnpm workspaces, where we’ll compare their strengths, weaknesses, and ideal use cases to Nx. See you soon! 😍
Conclusion
Our deep dive into Nx reveals a powerful yet nuanced tool. Its strengths shine in large, complex monorepos, where its structured approach, build optimizations, and cloud capabilities significantly streamline development.
However, it’s not a one-size-fits-all solution. Teams with smaller projects, unique workflows, or a strong preference for flexibility might find Nx’s conventions and learning curve to be a hurdle.
Understanding Nx’s inner workings, from its project graph and task orchestration to the role of the daemon and Nx Cloud, is crucial for making an informed decision. The real-world experiences of other developers, both positive and negative, shed light on its practical benefits and potential drawbacks.
Remember, the best monorepo tool isn’t the one with the most features, but the one that best aligns with your project’s complexity, your team’s workflow, and your overall development goals. Armed with the knowledge gained here, you’re well on your way to monorepo mastery!
Stay tuned for our upcoming in-depth analyses of Turborepo and pnpm workspaces.
Until we meet again in a new article and a fresh adventure! ❤️
Thank you for reading my article.
Want to Connect?
You can find me at GitHub: https://github.com/helabenkhalfallah
