Ember and Yarn Workspaces
Square has several Ember web applications, and Square Dashboard is the oldest and the biggest. The original commit was lost to the dustbin of history, but I’ve been told it started in 2011 as a simple Sales Reports app on Sproutcore 2.0 (the predecessor to Ember). Today, it’s an Ember 3.0 app with Reports, Customer Management, Invoicing, Employee Timecards, and much, much more. Almost 100 Square engineers touch the codebase each month, and we deploy to production daily.
Building so much functionality into a single application allowed us to move quickly with new products and bug fixes. Until late last year, we didn’t have another option—we were using a bespoke Rails-based build pipeline that didn’t understand Ember applications. But the size of the application is a heavy burden: build and CI times are very slow, and the codebase is intimidating to new and seasoned developers alike.
- Addons and engines allow us to modularize the codebase and create boundaries between logical domains, making ownership clearer and allowing us to share code between teams without creating a rat’s nest of dependencies.
- Engines support lazy-loading code, meaning our merchants won’t download code for a section of the application until they need it.
However, breaking up the application is not a panacea. If we simply moved code from the application into separate NPM packages like this:
… and install the addon and engine into Dashboard like this:
… developing code simultaneously across packages would be quite painful. By default, NPM and Yarn would copy files between packages. If you change a file in
sales-reports-engine and want to see it reflected in
dashboard, you’d have to completely reinstall the engine package.
What we wanted was a way to break up the app without hampering the development experience for everyone—especially as we encourage engineers to spend most of their time developing addons and engines, instead of adding to the main application.
One solution would be to use npm link, a tool to symlink packages together. (
yarn link does exactly the same thing.) Using it is fairly straightforward:
> cd sales-reports-engine
> npm link
success Registered "sales-reports-engine".
info You can now run `npm link "sales-reports-engine"` in the projects where you want to use this module and it will be used instead.
> cd ../dashboard
> npm link sales-report-engine
success Using linked module for "sales-reports-engine".
After running those commands, you’d see a symlink to the engine folder in the
This structure allows Dashboard to access files directly from your working copy of
sales-reports-engine during development.
However, this is insufficient if you want to see changes reflected across all three packages at once. What you really want is:
This arrangement would take numerous
npm link commands to construct, especially once we have more than three packages. And personally, I could never reliably get
npm link to work—the tool works correctly, but I kept swapping the commands or forgetting whether I had run them already.
Fortunately, there’s a better way! 🙌
Support for workspaces arrived in Yarn in August, 2017, and we’ve found the feature to be stable and easy to use since version 1.3.2. And as of Ember CLI 3.1 (now in beta), workspaces and Ember are best friends!
To set everything up, you’ll need to move your packages into a “workspace root” with its own
The contents of the workspace root
package.json are very simple:
Now when you run
yarn install anywhere inside the workspace root, Yarn will discover the dependencies between packages and hoist symlinks up to a top-level
There is no step #2! Most engineers working on Dashboard probably don’t realize this is even happening.
If you’d like to know more about how the packages find their dependencies in the top-level
node_modules folder, the Node docs explain the lookup algorithm.
What about in-repo addons?
Ember CLI’s original solution for simultaneously developing addons is in-repo addons, which have never required any
npm link shenanigans. We have a few reasons for not using them:
- In-repo addons cannot have their own test suites. Test files go into the host application. We want our tests to be as decoupled as our application code.
- We want to be able to publish certain addons for use by other Ember applications at Square. In-repo addons are not real NPM packages so they can’t be published on their own.
- Similarly, we want to allow teams to escape from the monorepo in the future if it makes sense for their product. Once their code is in a “real” addon with its own dependencies and test suite, they can move the files to another repository without much effort.
NB: Naming is hard—our “real” addons are inside the same git repository, so aren’t they “in-repo” addons? 🤯 To disambiguate the two patterns, we use the term “monorepo addon” for addons in our Yarn workspace.
Are there any gotchas?
Not really! You must be on Ember CLI ≥ 2.18 for Yarn workspaces and Ember to work at all (Edward Faulkner’s commit has a great explanation for why that is).
If you’re on Ember CLI ≤ 3.1, commands like
ember install will default to using NPM, but this is remedied by adding the
ember install ember-animate --yarn
(Our very own Timothy Park landed the patch to fix this in 3.1!)
Migrating to Yarn Workspaces
You’ll have the best luck if you follow this three-step process:
- Move your existing
yarn.lockfile from the original application folder up to the Yarn workspace root.
- Create the new
package.jsonfile next to the lockfile as described in the Yarn docs.
If you don’t move the original lockfile, Yarn will create a new lockfile and upgrade all your transitive dependencies. This could lead to unexpected breakage.
Switching our application to Yarn workspaces is one of the most quietly impactful decisions we’ve made in our journey to modernize the codebase. Instead of one gigantic application, we now have a (slightly) smaller app and over a dozen well-organized addons and engines that are much nicer to work with. If you have a monolithic application that may start to buckle under its own weight, I highly recommend giving Yarn workspaces a try.
P.S. One extra pro-tip:
yarn upgrade-interactive and
yarn outdated work fantastically inside of a Yarn workspace. 💯