Gzippin’ Javascripts

Published in

Everestate

10 min readJun 20, 2019

Introduction

This is the first time I’ve written an article about anything code-related. I’m publishing it because I hope it helps someone who finds themselves in a similar situation, trying to accomplish what should, in 2019, be a very simple task: serving gzipped Javascript bundles, generated and compressed using Webpack 4, hosted on S3, and requested via inline script tags output by the webpack-html-plugin.

About me: I’m a Javascript developer living in Berlin. Working with Webpack is something I've done out of necessity, and I'm still figuring things out along the way.

If you would like to skip the backstory, you can jump ahead to see the TL;DR solution.

Part I — the challenge

This particular issue has been in progress for several months. At some point last fall, I picked up a ticket related to page-speed optimisation, where results from a Lighthouse test suggested that we could improve our performance by serving compressed code to our users, decreasing the size of what they needed to download from the servers in order to visit our site. At the time, we were still using Webpack 3, and we had some compression middleware in place on the server, which had been implemented by someone else, and was possibly not really doing anything useful. After poking around for a couple of days on the ticket, I found this article, which pointed to the compression-webpack-plugin as the solution for implementing dynamic gzipping at build time.

The basic idea was to compress the files once at build time, rather than waiting to compress them at the time at which they were requested by a user.

The article’s suggestion looked like what we needed, but it required a minimum of Webpack v 4.0, and so it wasn’t possible for us to implement it yet. We tried a few other approaches, unsuccessfully, and ultimately decided to just configure S3 to compress our JS assets on-the-fly. It seemed like the obvious solution, and I can even remember it working…

Part II — the struggle is real

Just before the end of the year, we successfully upgraded Webpack to version 4.0+, and along the way, we had to change various plugins and optimisations to get things working with the new setup. One issue was that building the app locally in development took a long time, and hot reloading the app ended up crashing frequently.

These problems were mostly solved by adding mode: ‘development’ to the dev config and removing the hashing from the ExtractCssChunks plugin.

About a month ago, I decided to run another Lighthouse test and was floored by the result. Somewhere along the way, the size of the code had ballooned in size: it was BIG, it was CRAZY BIG! Other developers worry about bundles larger than 200 kB, and here we were, asking our users to download 24 MB in a single bundle. No, that’s not a typo. It was shocking.

And so I returned to the topic of how to reduce our bundle size, diving warily into our Webpack configurations, and crossing my fingers. I started with the BundleAnalyzer, a fantastic tool that lets you visualise your code. It gives you something like this:

The button on the top left expands a side panel that shows you the size the individual blocks, ranked by size. I could clearly see giant megabytes of code, most of which coming from our node_modules. My first action was to split the code into the main and vendor bundles, using the splitChunks optimisation and defining the two separate cacheGroups. This didn't decrease the overall size, but split a large block into two smaller blocks which could be downloaded simultaneously, as well as cached separately and more efficiently.

The issues at hand

There were two major issues that I needed to solve

to find a way to serve compressed versions of our JS bundles
to figure out why the bundle sizes reported by the BundleAnalyzer in development were significantly different (smaller) than those downloaded in production.

For the first problem — of how to serve Gzipped Javascript, I returned to that original article I had found last year. Now that we had upgraded Webpack we were hypothetically ready to implement the build-time gzipping, using the compression-webpack-plugin.

The article suggested two steps: first, to use the CompressionPlugin to gzip the assets at build time, and second, to add an Express middleware on the server in order to modify the request to point to the .gz files, and then to set the appropriate response headers for the compressed files.

// in webpack config file const CompressionPlugin = require(‘compression-webpack-plugin’);...plugins: [
  new CompressionPlugin({
    filename: '[path].gz[query]',
    algorithm: 'gzip',
    test: /\\.js$/,
    minRatio: 1,
  }),
],
// in server/index.jsapp.get('*.js', function (req, res, next) {
  req.url = req.url + '.gz';
  res.set('Content-Encoding', 'gzip');
  next();
});

This solution worked perfectly. Looking into our buildClient folder, I could see that Webpack generated the regular .js bundles, and then compressed them into .js.gz bundles, which were MUCH smaller. I was able to run the application locally, and see that the compressed files were downloaded, and everything worked as intended. Yay!

I created a PR with the changes and excitedly waited as it was deployed to staging. Alas, it didn’t work quite as intended — or rather, nothing changed at all: we were still downloading the regular, non-compressed bundles.

Why was it working locally but not on staging? The reason was the way in which the assets were being served. The html-webpack-plugin was generating markup that included inline script tags pointing to our asset bundles. Locally, these requests pointed to our server and were funnelled through our middleware as intended. On staging, however, these script tags pointed directly to S3, so the requests bypassed the middleware altogether.

Since we were using inline script tags for these assets, we didn’t seem to have a request object available that we could modify. Finally we had an idea: what if we could modify these generated script tags in order to have them point directly to the compressed assets, those which had the .gz extension?

Essentially, we wanted <script src="/static/vendor.js"></script>to become <script src="/static/vendor.js.gz"></script>

I googled around on the issue and found several other people inquiring into the same functionality. The issue had been reported, even marked as resolved, but we (and many others) did not see the desired output.

Eventually, convinced by this approach, but not finding our way through, we tried to solve this by modifying these script tags directly , which resulted in very ugly fix in which we were rewriting native toString method, in order to add the .gz extension:

// Please do NOT use this code, scroll to the end of the article!const chunkNames = flushChunkNames();
  const {
    js, scripts, styles, cssHash,
  } = flushChunks(clientStats, {
    chunkNames,
    before: ['bootstrap'],
    after: ['vendor', 'main'],
  });  const zippedJs = { 
    toString: () => js.toString().replace(/.js/g, '.gz.js'),
  };  res
    .status(getStatusCode(req.url))
    .header('Cache-Control', `public, max-age=${oneHour}`)
    .render('index', {
      build: '/build',
      RENDER_ON_SERVER: true,
      app,
      js: zippedJS,
      styles,
      cssHash,
      relayPayload,
      hash: fileHash,
      htmlAttributes,
      link,
      htmlWebpackPlugin: {
        options,
        files: { chunks: [] },
      },
    });

We ultimately got this working; the generated HTML file output by webpack-html-plugin contained the desired script tags with .gz extension appended. I also modified the express middleware for development purposes, so that it no longer modified the request object to point to the .gz file, but just set the response headers as needed.

// in server/index.js app.get(‘*.gz’, function (req, res, next) { 
  res.set(‘Content-Encoding’, ‘gzip’); 
  next(); 
});

Unfortunately, when we deployed to staging we discovered that all of the images were missing on the app, along with other errors in the console. The script tags were working and the Gzipped bundles were being downloaded, but they were not getting decompressed by the browser.

This makes sense, of course, because the staging requests were bypassing the middleware, as before, so the content-encoding headers were not being set. This meant the browser did not know what it was supposed to do with the compressed code it was downloading.

<script type='text/javascript' src='<https://static.everestate.net/vendor.99911a91ccf8015f5948.js>' defer='defer'></script><script type='text/javascript' src='<https://static.everestate.net/main.ffc9301a7cea6a8956b0.js>' defer='defer'></script>

We were compressing the files, and serving them successfully from S3, but we still needed a way to set the response headers. S3 itself did not provide us with an obvious way to do this. Finally, however, we came across the S3 Plugin, which allowed us to set the Content-Encoding headers at build time, just before the compressed files are uploaded to S3.

const S3Plugin = require(‘webpack-s3-plugin’);// added to webpack config plugins array
new S3Plugin({
  exclude: /.*\\.html$/,
  s3UploadOptions: {
    Bucket: process.env.ASSETS_BUCKET,
    ContentEncoding(fileName) {
      if (/\\.gz/.test(fileName)) {
        return 'gzip';
      }
    },
  },
}),

Everything looked good, we deployed on staging, and could see that the compressed bundles were being downloaded and properly decoded by the browser. Problem solved!!

At this point I moved on to solving my second challenge, which was figuring out why our production bundles were SO MUCH LARGER than the ones seen in development. I was already setting Webpack’s mode: production, which enables further optimisations automatically, so the expected result would have been SMALLER bundles, but in practice they seemed to be about 5-fold larger in production.

I went line-by-line comparing our development and production configurations, but I couldn’t find anything that popped out. The only thing that was different, other than the mode, and the chunk hashing, and other things that I could clearly rule out was the devtool. Indeed, we had made some changes to the source maps used in development and production at various points along the way, and apparently, we had chosen an inappropriate sourcemaptool 'eval-cheap-module-source-map' for our production environment.

There are a number of different sourcemap tools available, and each is optimised for different use cases and environments. In general, you have a tradeoff to make between more complete source maps vs speed loading and running your application. Another concern is how much of your codebase you wish to expose in production. These different tools can have substantial effects on bundle size, so please do your research in advance and be sure to select an appropriate option.

In case you are interested, after researching these devtool options further, we decided to go with the regular 'source-map' for our development and stagings environment, and the 'hidden-source-map' devtool for live production (to enable accurate stack traces for errors, without exposing the entire code to the world).

I created a PR with these changes, excited about finally solving the riddle, and happy that it could be included in the same release with the other gzipping performance optimisations. A few minutes later, just as we were finalising the release, we stumbled upon an unpleasant surprise — a colleague reported major usability issues, site-wide while testing on Firefox.

Part III — the gotcha

The culprit, of course, was the gzipping again — it seems that Firefox handles these files differently — if it sees a file with a .gz extension, it assumes that the user wants to download (and save) the compressed file as-is. So to keep Firefox happy, we need to serve files with a regular .js extension, but still adding the content-encoding: gzip headers on the response. Thank you, Firefox, for being such a stickler... Sigh.

Once we got past the frustration inherent to cross-browser testing, reimplementing our solution in the way that suited Firefox did enable us to remove the hacky rewrite of the .toString method implemented earlier, reverting all changes made to the output of the html-webpack-plugin, but still using the S3 Plugin to set the appropriate response headers.

We did have to modify our settings for the Compression Plugin, so that it would output files with their original extension, and not append the .gz (see final result below).

Of course, there was one last gotcha hiding — when we merged the changes to staging we didn’t quite see everything magically fall into place. The downloaded files had the correct .js extension, and the correct Content-Encoding: gzip response headers, but they were STILL COMPRESSED. WHY?!

Way back at the beginning of this article I mentioned that we had turned on automatic GZIP compression functionality for all JS served from S3. This had not been doing anything before as we hadn’t been sending the correct request headers (and later because we were requesting .gz files instead). But now, for some reason (perhaps based on the response headers we were setting via the S3 plugin), it was working, and was compressing our already-compressed bundles: double-compressing our JS!

The solution

Upgrade Webpack to version 4+
Enable Build-time gzipping using the Compression Webpack Plugin

// in webpack config file const CompressionPlugin = require('compression-webpack-plugin'); 
... plugins: [ 
  new CompressionPlugin({ 
    filename: '[path][query]', 
    algorithm: 'gzip', 
    test: /\.js$/, 
    minRatio: 1, 
  }), 
],

set Content-Encoding: gzip headers for .js files via Express middleware, for serving assets locally

// in server/index.js app.get('*.js', function (req, res, next) { 
  res.set('Content-Encoding', 'gzip'); 
  next(); 
});

set Content-Encoding: gzip headers for .js files via S3 Plugin, for serving assets via S3 in production / staging.

// in webpack config new S3Plugin({ 
  exclude: /.*\.html$/, 
  s3UploadOptions: { 
    Bucket: process.env.ASSETS_BUCKET, 
    ContentEncoding(fileName) { 
      if (/\.js/.test(fileName)) { 
        return 'gzip'; 
      } 
    }, 
  }, 
}),

Use an appropriate devtool (source-map / hidden-source-map / none) in production environment, for smallest possible bundles.

Summary

We successfully reduced our code bundles down to two chunks at 551kb and 293kb. Having started at 24Mb, this represents a 28-fold decrease. Ideally, we will continue our work to get these down to under ~200kb each, it’s a work in progress :-)