Getting the most out of Webpack(er), Part 2
Optimizing our bundle for faster page loads
In Part 1, we looked at the Webpack Bundle Analyzer, the primary tool we use at Flatiron School to optimize the JavaScript bundles powering Learn.co:
In this part, we’re going to discuss five rules that helped us optimize our JavaScript performance with Webpack:
- Only import what you need
- Don’t implicitly trust your package manager
- Keep dependencies as up-to-date as possible
- Proactively strip out what you don’t need
- Split aggressively… but also thoughtfully
Setting the stage
We’re going to focus on optimizing Learn.co’s commons chunk — the bundle that contains most of our vendor JavaScript, such as React, Xterm.js, and the Ace editor. We split that vendor code out from the rest of the JavaScript we’ve written in-house with Webpack’s CommonsChunkPlugin¹ in order to minimize the amount of JavaScript our users have to download on return visits to the site.
As we optimize the commons chunk, the metric we’ll use to track our progress is gzipped kilobytes. At the start of our journey, the commons chunk weighs in at a hefty 1,216 kB.
Seeing double
When we first fire up the Bundle Analyzer, our first issue immediately jumps off the screen. Either someone’s spiked the office water cooler, or we need to go get our eyes checked:
If you can’t tell from the impossibly tiny text in the above GIF (we’re software engineers; not graphics experts), that’s:
- Two copies of Moment.js, including all of its locales.
- Two copies of jQuery.
- Two copies of Backbone.js.
- Three copies of Underscore.js.
We don’t have many math majors on the team, but by our unscientific count that’s approximately 2–3 times more packages than we need.
We’ll start with the double jQuery, which leads into our first Big (if obvious) Takeaway™:
1. Only import what you need
It seems obvious, but in a large application with hundreds or thousands of JavaScript files it can be difficult to keep track of which import and require statements are necessary, especially over time. Properly-configured code quality tools like ESLint can help alert you to unused imports, but they aren’t perfect.
When we migrated from Browserify to Webpack, we removed all of the traditional JavaScript manifest files required by Rails’ Sprockets-powered asset pipeline. jquery_ujs, Rails’ unobtrusive JavaScript adapter, was listed in each of the deleted manifests. We weren’t sure whether we still relied on any of its functionality, but one of the primary goals of our switch to Webpack was to minimize the waves created by the already-wide scope of the transition. We decided it was safest to maintain parity by adding the jquery-ujs package as an import everywhere it’d previously been available via a Sprockets manifest.
When we added the package, Yarn looked at jquery-ujs's dependencies, saw "jquery": ">=1.8.0" as a peer, and checked whether it already had an identical version expression:
When it didn’t find an exact match, Yarn resolved >=1.8.0 to jQuery 3.3.1:
And thus were our beautiful twin jQueries born.
Ultimately, we undertook a more thorough search and determined that we could remove jquery-ujs entirely, remediating the problem. However, the fact that it was an issue at all leads into our second Big Takeaway™:
2. Don’t implicitly trust your package manager
Yarn is a very cool piece of software created and maintained by very smart people. Yarn’s sole purpose isn’t to collaborate seamlessly with Webpack; it’s to manage packages. It’s a reasonable assumption that when we add a brand new package into an ecosystem full of old packages we’d prefer to use the latest and greatest versions of that new package’s dependencies. It’s an equally reasonable assumption on Yarn’s part that it’s probably best to let sleeping dogs lie — just because a sub-dependency could be resolved to a higher version doesn’t mean it must, and it’s better to wait on proaction from the user to do so.
However, the upshot of Yarn being so darn reasonable is that we were left packaging up two copies of jQuery in our commons chunk for a short while. We’d ducked action in the first instance by simply removing the problematic package, but we’ll have no such easy path out with the second: Moment.js.
A couple months ago, we bumped the version of our Moment.js dependency from 2.10.x to ~2.18.1. Prior to the bump, our yarn.lock file looked like this:
And post-bump:
Even though >= 2.9.0 is a clear superset of ~2.18.1, Yarn decided to play it safe and left good enough alone. Since the resolution of >= 2.9.0 to 2.10.6 only exists inside of the yarn.lock file, we could try deleting the offending lines (7–9 in the above gist) and re-running yarn. Upon not finding an existing resolution for "moment@>= 2.9.0", Yarn grabs the latest version, 2.22.2. Uh oh!
Ah, yeah, we forgot about that pesky warning at the top of every yarn.lock file:
# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.Trying to manually bend lock files to our will is a risky prospect, especially if the package in need of bumping is tangled in a twisted web of peer- and sub-dependencies. And, anyway, Yarn already provides a nice API for us to accomplish exactly what we want.
If you’re feeling adventurous and don’t have to operate within a codebase that contains some hasn’t-been-touched-in-years, keep-everything-locked-to-the-exact-versions-that-work-and-don’t-change-a-thing legacy code, try out yarn install's --flat option, which will walk you through the process of selecting a single version per dependency. We do have to maintain a fairly sprawling Backbone / Marionette / Alt / jQuery-driven chunk of code, so we’ll take the scalpel approach and set a single, targeted dependency override for Moment:
Et voilà!
We’re now down to a single copy of Moment.js in our commons chunk, and, together with the jQuery de-duplication, we’ve managed to trim the chunk down to 1135 kB, a full 6.7% decrease! We’ll come back to deal with the duplicate Backbones and Underscores still floating around, but first let’s take a brief detour to talk about the importance of updating dependencies.
3. Keep dependencies as up-to-date as possible
Upgrading Moment.js caused an unfortunate temporary regression in bundle size, but, in general, keeping everything up-to-date should be a top priority. For an active, well-maintained package, upgrading can prevent historical dependency cruft from creeping into your bundles. For an example, let’s turn back to the Bundle Analyzer and take a look at the updated commons chunk:
What even is all that? elliptic is a library for performing elliptic curve cryptographic operations in JavaScript. readable-stream is an extraction of the Streams API out of Node.js’s core. diffie-hellman? hash.js?! browserify-sign?!? node-asn1?!?!
It turns out that version 0.13.6 of the Sails JS client, which powers our legacy real-time chat on Learn.co, depends on Request, an HTTP client for Node.js. The dependency was changed to a dev dependency in the 0.13.7 release, and the call to require('request') was removed from the distributed code.
However, we have a strict version requirement that has prevented us from receiving even the 0.13.7 and 0.13.8 patch releases:
We use Sails in a number of places across our monolith, and Webpack dutifully heads further down the rabbit hole with each additional nested require. It’s node_modules all the way down! Sails 0.13.6 requires Request ~2.34.0, which requires… well… a solid chunk of the Node.js ecosystem.
After a thorough QA process, we were able to bump our Sails dependency to 0.13.7 (and then 0.13.8 for good measure). By relaxing our strict dependency and jumping up a single patch version, our commons chunk shrank by 11%, down to 1010 kB. Once again for the people in the back: absolutely-useless-in-the-browser Node.js-land cruft made up over a tenth of our entire commons chunk.
Keeping packages updated is awesome for a whole host of reasons, and, at a very basic level, it’s just more fun to play with shiny new toys. However, don’t go about blindly upgrading every package under the assumption that it’s the one true key to reaping untold bundle optimization riches. It’s scary out there in the Wild West that is the JavaScript package ecosystem, and sometimes we need to take matters into our own hands. For an example, let’s return to our old friend Moment.js and discuss a corollary of the very first piece of advice:
4. Proactively strip out what you don’t need
Webpack (via Webpacker, in our case) has a ton of plugins that can be used to optimize your output bundles in various ways. Some of them, such as the UglifyjsWebpackPlugin, have an extremely broad purview, mangling code left and right across the entire codebase. Others, such as the IgnorePlugin, can be used in a much more targeted manner, only affecting a well-scoped sliver of the mass of code processed by Webpack. That targeted approach comes in handy for dealing with Moment’s locales, an ever-growing series of extensions to the core library that localize date-time representations for different audiences around the world.
Not all libraries are set up for seamless integration with a modern, tree-shaking build tool like Webpack. When Webpack arrives at an import moment from ‘moment' statement and starts processing the library, it loads every single file in the ./locale/ directory. By the time Moment makes it into our commons chunk, the core library has been mangled and compressed down to 16.4 kB, but that child directory housing locales weighs in at 46.1 kB, almost three times the size. At this time, we don’t perform any date-time localization on Learn.co, so every single locale file is unused.²
The bloat added by Moment’s locales is a common gripe, especially within the bundle size-conscious Webpack community. It’s so common that one of the two sections on the IgnorePlugin in Webpack’s official documentation is dedicated to how to use the plugin to strip out Moment’s locales. And that’s exactly what we’re going to do:
The first argument passed to the IgnorePlugin tells it to ignore any request for an asset (via a require() or import statement) matching ./locale, and the second argument scopes it down to only ignore matches that originate from a module directory ending in moment.
When we fire up the Bundle Analyzer again, look how much smaller Moment.js gets:
The commons chunk shrank by another 4.6% to 962 kB, and, short of massive refactors to remove legacy package dependencies, we’re running out of slim-down targets. But even though there aren’t any packages left to bump or unused imports to ignore, there’s one final optimization to make that will have as much impact as the previous four combined:
5. Split aggressively… but also thoughtfully
Our commons chunk setup should:
- Consist of JavaScript code that’s used in multiple places across the app.
- Minimize the amount of code browsers have to download by preventing duplicate code from being packaged up in multiple bundles.
- Avoid including modules that are frequently updated in order to take advantage of browsers’ asset caching abilities.
The first bullet point is a bit more complex than it may seem at first blush. We have a large vendor asset that is version-locked. However, it’s only used in a single place in the application. How do we weigh whether to include it in the commons chunk? The answer differs if that single place is on a high-traffic page versus a lower-traffic feature that perhaps requires some user action to activate.
When we first moved to Webpack, we set up our CommonsChunkPlugin to split out vendor assets into a separate chunk exactly like this example from Webpack’s documentation:
minChunks' property a function.There are a few issues with that setup, but the one we’re going to focus on now has to do with the way this setup decides which code belongs in the commons chunk and which does not.
The code above says that every imported module will be passed to the minChunks function during Webpack’s build process. If the passed-in module lives in the node_modules directory, it will be moved into the commons chunk. Simple as that.
That setup is a bit naïve, as it doesn’t distinguish between a vendor module that’s used in a single, isolated place within the application and a vendor module that’s used in every single bundle. It also doesn’t take into account non-vendor modules that rarely change and are used widely across the app.
As it happens, three of the four largest modules in our commons chunk fit that first narrative to a T. The c3 library and its chief dependency, d3, are, respectively, the third and fourth largest modules, and they’re used to graph student progress in a single, admin-only interface. Not only is the page restricted to admins, but the graphs aren’t even visible on initial page load — the admin must perform an action for them to appear. It’s legacy code that will likely be removed in the not-too-distant future, but it makes little sense to include both libraries in the commons chunk that’s downloaded by every single site visitor.
While it sees significantly more usage than c3 and d3, it’s equally difficult to justify including the brace module in the commons chunk. It accounts for over a third of the commons chunk’s size on its own, checking in at nearly five times the size of the second largest module. brace is the editor component of the in-browser IDE, which is only available on lesson pages under certain conditions. And, just like the graphing libraries, the IDE is never visible when the page loads — students must click a button to open it.
Because both the graphing libraries and the IDE editor package require users to take an action before they are needed, they’re perfect candidates to be split out into separate chunks that can be loaded only when they’re needed. Let’s adjust our CommonsChunkPlugin setup a bit to create three separate commons chunks — one for the IDE, one for the Organizations app (where the graphs live), and one for everything else in node_modules. The ‘everything else’ chunk has to be defined first:
The ordering of the other two chunks doesn’t matter, but they must come after the main chunk because they’re only going to extract modules that have already been bundled up in the main chunk. We’ll do the organizations chunk first:
Note: tablesorter is another vendor library that’s only used within the organizations app, so we’ve split it off into the organizations-commons chunk.
And then the IDE chunk:
Note: react-ace is the actual editor package that brace wraps, and lodash.isequal is a deep comparison function that react-ace depends on. Both were logical choices to package up with brace in the ide-commons chunk.
Before spinning off those specialty commons chunks, a student’s browser would have to download a 962 kB commons chunk the first time the student navigated to the site. After the change, that figure is down to 490 kB, a 49% drop, with the IDE commons chunk weighing in at 351 kB and the organizations chunk at 122 kB.
Final notes
Relative to our initial starting place of 1,216 kB, we’ve managed to cut the single largest piece of JavaScript we load synchronously on initial site visits by 60%. That’s 60% less JavaScript that a new visitor’s browser has to download, decompress, parse, compile, and execute prior to the site becoming usable:
Here’s a complete gist showing everything we’ve talked about laid out in Webpacker’s environment.js file:
And here’s a final look at our three commons chunks via the Bundle Analyzer:
Thanks for reading!
P.S.: Want to work on a mission-driven team that loves ice cream and optimized JavaScript build pipelines? We’re hiring!
Footnotes
- Deprecated in Webpack 4 in favor of the SplitChunksPlugin.
- Even if we only supported a small subset of languages, we could use the ContextReplacementPlugin to whitelist the subset and discard everything else. Or, if we wanted to support every locale, we could probably send up an array of the choices and then load the actual locale files as they are requested by the user.
To learn more about Flatiron School, visit the website, follow us on Facebook and Twitter, and visit us at upcoming events near you.
Flatiron School is a proud member of the WeWork family. Check out our sister technology blogs WeWork Technology and Making Meetup.
