webpack optimization — A Case Study

Since we open sourced part of the Electrode platform from @WalmartLabs, we’ve had a lot of contributions to our NodeJS and ReactJS archetype modules, including bug fixes, new features, and updating to webpack 2.0. To take advantage of these, we are migrating our internal apps to use the OSS versions. The first app we did was HomePage that serves the http://www.walmart.com landing page. In this blog I will talk about some issues we uncovered using our latest archetype in a real world application, and the solutions we implemented.

Updating Electrode Archetype

It was relatively straightforward to convert our HomePage app to use our latest OSS archetype since we’ve been keeping our internal ones mostly in sync on minor features. The process involved updating the npm packages and a few require calls in the code and it was good to go. After that, the usual smoke tests all seem good. The dev mode started without error and a copy of walmart.com was served locally. The production build also completed without a hitch. So far so good and it also reinforced the design goal and flexibility of our archetypes — all the new features available to the app and webpack 1.0 to 2.0 for a production app just by updating dependencies.

We submitted a PR for the changes and was hoping it would go through the CI and regression tests. However, our HomePage tech lead Arunesh Joshi came back with a few issues after reviewing it. There were a few minor things such as linting that was fixed quickly in the archetype, but what Arunesh noticed was that the bundled CSS and JS files’ sizes increased significantly, by more than 30%. It was time to do some investigation.

Sheng Di, an Electrode team member at @WalmartLabs, did some investigative work on the issues Arunesh raised and she opened the PR here to fix the lint issue. She found that the increased CSS size was because we erroneously removed the optimizing plugin when we updated to webpack 2.0. She opened this PR to fix it. On the JS bundle size, she checked the webpack stats and found that some modules coming from node-libs-browser has increased in size in webpack 2.0 vs 1.0. She also found a related question on Stack Overflow about it. A very good lead and time to dig deeper.

Learning NodeSourcePlugin

Before we try to find out why node-libs-browser code has increased in size, the more puzzling question was, why were we even pulling in code from that? To find out what JS code requires that, a simple recursive grep for the string node-libs-browser on all the JS files, including those under node_modules, yielded the answer. Below is the command I used:

$ find . -type f -name "*.js*" | grep \.js[x]*\$ |  grep -v \/test\/ | grep -v \/example\/ |  grep -v \/dist\/ | xargs -n 20 grep node-libs-browser | less

The result:

./node_modules/webpack/lib/node/NodeSourcePlugin.js:const nodeLibsBrowser = require("node-libs-browser");
./node_modules/webpack/lib/node/NodeSourcePlugin.js:                            return require.resolve(`node-libs-browser/mock/${module}`);
./node_modules/webpack/lib/node/NodeSourcePlugin.js:                            return require.resolve("node-libs-browser/mock/empty");

Of all the JS code, only a file NodeSourcePlugin.js from webpack references node-libs-browser. Why is that causing it to be bundled into our app? This led me to look into webpack’s docs and code.

This is what I found out:

• webpack’s target config is default to web
• The web target by default applies an internal pluginNodeSourcePlugin, which automatically makes NodeJS builtin modules available for the browser through node-libs-browser.

Getting Down to the Code

The first thing I tried was to remove NodeSourcePlugin. However, after going through webpack’s docs and code, there didn’t seem to be an easy way to disable that with a single switch. I opened an issue here and after some discussion with webpack’s Tobias Koppers, we settled on allowing webpack’s node option to accept a false config.

With NodeSourcePlugin disabled, our code that depend on it all failed and I was able to pinpoint among our hundreds of components, which one to look into.

I found out that a few components that have server side only behavior implemented it through the runtime flag provided by exenv, like this:

import { canUseDOM } from "exenv"
function foo() {
  if (!canUseDOM) {
    const crypto = require("crypto")
    // do something with crypto
  }
}

The idea is simple, if code is not executing on the browser, then do something on the server side only using the crypto module. Unfortunately, webpack only does static analysis to find out what code needs to be bundled. That means NodeJS crypto is included even though it’s only needed for server side. And this has gone unnoticed for a while even when we were using webpack 1.0.

The Solution

To fix the issue, I extracted all the server side only code into another npm module. In its package.json, I pointed the browser field to a dummy JS file that has empty functions and the main field to the real implementations. The end result was a reduction of 100K in the optimized bundle.

We still have a few other components that reference NodeJS builtin modules like path and util. Even though they are small, I still plan to refactor them so we can stop relying on NodeSourcePlugin. I will turn it off by default in our next major archetype release to prevent any code from pulling in huge server side only code. It’d be nice if NodeSourcePlugin allows switching NodeJS builtins to a whitelist only mode instead of all enabled by default.

Conclusion

Due to webpack’s default inclusion of NodeSourcePlugin, there is a side effect that makes your server side only code to be bundled if they are only turned off by runtime flags. This is something hard to track when you have large projects with many teams working on hundreds of components. We had this issue and our bundle has been including 100K of minified code unintentionally. I fixed it using webpack’s support of the browser field in package.json. I recommend that you double check this in your app also. I plan to turn off the plugin by default in our archetype modules to make this issue easier to catch.

Shout outs

Huge thanks to @WalmartLabs HomePage tech lead Arunesh Joshi for always being the first to help us with new features, Electrode team member Sheng Di for doing the excellent work for the platform and this update, and of course the amazing webpack contributors and specifically Tobias Koppers for everything.

And of course Dave Stevens for reviewing this.