Creating a module bundler with Hot Module Replacement
As a developer, I use a lot of libraries and tools developed by the community in my day to day work. Most of the time, I am not interested in how they actually work as long as they get the job done. There are a lot of tools like these — code linters, module bundlers and sourcemaps to name a few. I sure do understand them at a high-level, but I don’t know their internals. If I were to write my own version of a module bundler I wouldn’t know where to start. This post is my attempt at understanding one such tool and implementing a (over) simplified version of it from scratch.
One of those things that has always intrigued me is live reloading. It was amazing to see what LiveReload could do a few years back. I mean editor on one display and browser refreshing on the other display whenever you save files — it was magical. Then there was BrowserSync with its synchronised browsers idea. And now it is HMR — why refresh the browser when you can “hot replace” the code.
How it works
I started looking for hot module replacement related posts and came across this Stack Overflow answer (which is now added to webpack docs) which explains the various things in play to get it to work. App code, webpack compiler, modules implementing HMR API and HMR runtime — all work in tandem to achieve hot module replacement.
Here is what happens every time you make changes to your code if you have HMR configured in webpack:
- webpack rebuilds the changed modules
- Sends a signal via a websocket connection to HMR runtime running in the browser. HMR runtime is added to your app bundle when you enable HMR via webpack configuration.
- Upon receiving this signal, HMR runtime makes a request to webpack server to get a list of updated modules. Response to this request is a JSON containing a list of changed modules called an “update manifest”.
- HMR runtime then downloads the updated modules (update chunks) via JSONP and applies the updates
The downloaded update chunk is a call to “webpackHotUpdate” function with the updated modules passed in as arguments. This function is immediately called when the update chunk is downloaded and the modules in the browser are updated with the latest updates. Here is how the update chunk looks if you make changes to a stylesheet.
All that was said till now in this post holds good theoretically, but I wanted to see how challenging it will be to apply this knowledge in creating a module bundler of my own which is capable of doing hot module replacement.
Writing a Module Bundler
We will be creating a module bundler which will bundle our app code written as CommonJS modules to make them run in the browser (something like browserify). The sample app code is split across two files — app.js is the entry file which updates an element’s (#message) text in the page, message.js provides the text.
So how do we takes these files and create a bundle which runs in the browser? We are going to rely on a few packages to do the heavy lifting for us.
A good way to understand how a tool works is to look at it’s dependencies. I found these two interesting packages in the dependencies of browserify.
- module-deps — It takes an entry point file as input, recursively traverses through all of it’s dependencies and outputs a JSON. The JSON contains all the files and their dependencies starting from the specified entry point file.
- browser-pack — Takes the JSON output from module-deps as input and creates a bundle capable of running in the browser.
Let’s start with an index.js file to create our module bundler. Here is how the implementation of our module bundler looks so far. It does the following:
- Accepts the entry file path via command line
- Provides the entry file path to module-deps and gets the JSON output
- Feeds the JSON output from module-deps to browser-pack to generate the browser bundle
- Writes the final bundle to dist/bundle.js
We need to customise the bundle a bit to add a function which will update modules whenever updates are detected — think of the “webpackHotUpdate” function I mentioned earlier. We will get to that shortly.
Watching for changes and notifying the browser
Now that we have a functional module bundler, the next step is to watch for changes to any of the files and notify the browser about them. We will be doing the following things to achieve this:
- Listen for file system changes
- Recreate the bundle with the latest changes
- Notify the browser of the changes using a websocket connection (socket.io)
- Listen for messages through the websocket connection from the browser
Here are the changes to index.js which does items 1, 2 and 3 above. I have modified the processFiles function to invoke a callback, added fs.watch to watch for file changes inside “app” directory and send a message using websocket connection.
For the next step, we need to add a couple of scripts to the page running the application to listen for these websocket messages. Here are the contents of index.html which runs the app and hmr.js which contains code to listen for websocket messages. Notice that Socket.IO library and hmr.js are added to index.html.
At this point if you open http://localhost:3001/ in the browser and make any changes to files in the “app” directory, the browser will reload and run the app with the latest changes.
Hot update changed modules
Now that we have the infrastructure in place, we can look at hot replacing modules. Here is a list of things that need to be done:
- Create an express server end point, /hot-update, that will accept the file or module name as a request parameter and respond with the latest code for the module wrapped around a JSONP callback function (let’s call the function hotUpdate)
- Modify bundle output from browser-pack to include the JSONP callback function, hotUpdate, mentioned in step 1 above
- Send a request to /hot-udpate whenever a file change message is received via the websocket connection
Let’s start by defining the /hot-update endpoint. If you remember, we start the bundling process by feeding the entry point file to module-deps which returned all the modules and their source code in a JSON format. We can store this JSON in-memory and use it to retrieve the module’s source code while responding to the endpoint request. If you are curious, webpack does something similar with the help of an in-memory file system, memory-fs. I have added the following code to index.js which implements the endpoint.
We can move on to adding the hotUpdate function to the bundle output. This function will replace an existing module running in the browser with the latest changes supplied by /hot-update. Once the module is replaced, it will bootstrap the application by invoking it’s entry point module. Here is the function:
The final bundle is nothing but an IIFE which accepts all the modules passed in as a map. When an update is received, the hotUpdate function just replaces the updated module in the module map with its new defintion. Once the module is updated, it calls the app’s entry point so that the app bootstraps again.
Now that we have the function, this needs to be added to the bundle output so that it can be called whenever we receive updates. As we use browser-pack to create the bundle, we can accomplish this easily supplying an option called “prelude” to it. prelude is nothing but a “prefix” like code which will be added before the app code when creating the bundle. You can find the default prelude provided by browser-pack here. I have added the hotUpdate function defined above to the default prelude.
The module bundler is now close to completion, we just have to kick start hot module replacement whenever changes are made. To do this, we make a JSONP request to the /hot-update endpoint defined above, once we receive a “file-change” message via websocket (remember, we were listening to this message before to reload the browser). To make the request, we add a new <script> tag with it’s src attribute set to “/hot-update?moduleId=<id>”. I have modified the websocket message handler function, which was reloading the browser on file changes to make this request.
We have everything we need to test our module bundler (finally!). You can see the text in the page updating when I changed the text in message.js.
HMR API in webpack
Updating modules has a lot of side effects, which I have conveniently chosen not to worry about in this blogpost for the sake of brevity. This is where the HMR API methods defined in webpack come in. These methods can be implemented by modules to handle various side effects when performing an hot update. For example, dispose function can be implemented to perform clean up operations in a module before it is updated. The API also allows the developers to choose whether modules can accept updates with the help of accept and decline functions.
This was my attempt at understanding how module bundling and hot module replacement work and I learnt quite a few things while doing this exercise. I can also better appreciate the need for webpack’s various hot module replacement related API methods. While the idea looks simple, there is a lot of brilliance in play to get something this intricate to work in the scale of webpack.
If you have come this far, thanks for reading. You can find the source code for the sample app here.