From Grunt + Bower to Webpack, Babel and Yarn — Migrating a legacy build system

The build system that I had inherited for the International Cancer Genome Consortium’s Data Portal was fairly modern when the project first started in 2013, but hasn’t kept up with new developments in the years since.

The legacy build stack

• Grunt, bower, and usemin to concat our .js and .css files

The updated build stack

• Webpack with Babel and yarn

I’m generally hesitant on changing things that work, but in this case the technical debt was legitimately hurting our productivity.

Motivations:

Maintainability and Code Quality

We had some very large files.

I believe this was due to the high friction in creating new files

• Create a new file
• Add it to index.html and via a script tag (making sure to find the correct relative filepath first)
• Add the karma config (making sure to find the correct relative filepath here as well, which is different from the other path)

Sticking a function in an existing file is frictionless. That’s probably how we ended up with so many files with a thousand lines of code.

Having several Angular services or controllers that are each a few hundred lines long all in a controllers.js/services.js/directives.js makes maintenance rather painful. Instead of switching between descriptively named tabs, a developer would spend a lot of time scrolling up and down through hundreds of lines of code, taking care not to overshoot.

Things that weren’t Angular modules (mostly functions) also couldn’t be modularized unless they were stuck onto either the global variable or an Angular service.

Development Efficiency

• The old build system couldn’t use libraries from npm.
  An increasing number of packages are not available on Bower. Being able to leverage the npm ecosystem would allow us to easily pull in tools that can speed up development or help prevent bugs. (update: bower is now deprecated, with a notice recommending webpack and yarn instead)

Size of ecosystem

Here are the download trends for Babel, Webpack, Grunt, Bower, and I threw Gulp in there too for good measure.

Download counts obviously aren’t everything, but they are an indicator for the size of ecosystems. Larger ecosystems mean fewer features you have to write yourself (because someone already created it), and fewer bugs you have to debug yourself (because someone else ran into the same thing before and already found a solution)

Performance

• The legacy build system generated one 5MB bundle, 1.5MB of that was for a library used on two pages that most users are unlikely to visit. Being able to split that chunk out and defer loading it until needed saves unnecessary transfer and parsing.

Developer Happiness

“Arrow functions / destructuring / spread operators make us happy” is rarely sufficient justification for expending large amounts of time.

It is a great side effect though 😉

Steps

<script> tag to ‘require’s

One of the first things to do is to change all of the script tag imports <script src="*"></script> into requires. We had 169 of these script tags so we definitely didn’t want to do this manually.

regexer came in handy. Using a pattern of <script src="(.+)"><\/script> and a replace string of require('$1');, our tags were easily moved from our html file into our entry file for webpack.

Fix what breaks

Many of the vendor files we’re using are expecting dependencies to be in the global scope, and also don’t export anything since they expect themselves to be in the global scope. We also had a vendor lib that omitted variable declarations and uses implied globals, which now cause a ReferenceError: x is not defined error.

Luckily, Webpack already has solutions for this in the form of expose-loader, imports-loader, and exports-loader.

Here’s an example:

Let’s break that down

• imports?this=>window&outerRadius=>undefined
  The module is expecting itself to be in the global scope, so we have to import this=>window, setting the value of this to window for this module.
  It’s also assigning to a variable outerRadius without declaring it, so we need to declare the variable first and assign it as undefined.
• exports?donutChooserD3
  The module does not export anything, this yanks donutChooserD3 out from the scope of the module and sets it as the exported value
• expose?donutChooserD3!
  This sets the variable donutChooserD3 onto the global scope since there are other modules that are expecting it to be there.

More info on shimming modules with Webpack

There was one especially egregious library that had 104 objects/functions that needed to be exported into global. To get that list of the 104 names we needed to add to the “imports” string, we pasted the contents of that library into the console for an about:blank page and ran this:

This gave us the list of all the globals that aren’t native (i.e. were created by the library).

We saved the list of implicit globals and the list of things to export, and created the require string

For the curious, this generates a require string of:

You don’t actually need to read this

For maintenance reasons, we didn’t want to check this generated string in, and since require strings couldn’t be dynamic, in our code we require in require(process.env.GENOME_VIEWER_REQUIRE_STRING)), and then use Webpack’s DefinePlugin to replace process.env.GENOME_VIEWER_REQUIRE_STRING with the value exported from shims/genome-viewer.js.

Code splitting

Splitting out a library into a separate deferred bundle is as simple as wrapping require.ensure([], require => { /**/ }) at the very top of directive link functions where the library is used. Since we’re using an arrow function, context still remains the same and the code being wrapped shouldn’t need to be touched, the directive just takes a bit longer to initialize

Next steps

Migrate from lodash@3 to lodash@4.

It’s unfortunate that there doesn’t seem to be a completely painless way to upgrade from v3 to v4.

lodash-migrate, lodash-codemods, and eslint-plugin-lodash all help, but still require some level of human effort that’s relatively higher than dropping in a library. (update: lodash-backports comes very close, it backfills the removed aliases and functions and should greatly ease migration pains)

Still, the benefits of having access to some of the new utility functions without having to separately install each package outweigh the cost of running and potentially cleaning up after a codemod.

Migrate away from Bower.

We’re currently using both Bower and npm, with new dependencies using npm.

We should be able to migrate most of the packages still using bower without issue, but there are a few libraries whose versions may be so old that they aren’t available on npm, or they may have been abandoned. For those we may either internalize the dependencies (a somewhat icky path of least resistance), upgrade to the versions that are available on npm (will need to test for regressions), or publish to npm ourselves.

Update: It is done! The merged pull request — Remove bower, upgrade lodash v3 to v4

Summary

Build systems are important, but it’s also easy to get carried away, either by spending too much time getting everything “just right”, or making it overly restrictive and opinionated. It’s important to keep in mind that working on the build system is only incidental to working on the actual project, and that a good build system should get out of your team’s way, ideally allowing everyone to forget that it exists at all.

Thank you for reading! Feel free to tweet at me @CheapSteak, and you might also be interested in my article on Quick and Dirty tricks for debugging Javascript🕵️, and Using AngularJS components in React.

Sign up for my newsletter to receive new articles in your inbox

Originally published at softeng.oicr.on.ca on January 3, 2017
Edited with updates September, 2017