Concurrent Webpack Builds and Webpack's Node API

I've been working with Webpack for the last month, digging into its deepest functionalities and trying to improve my life trough this useful tool.

Is very common that applications grow fast, and like mentioned in my previous article, we don't want to lose time while building applications and delivering code not only in development but in production phases.

This time I want to show how to configure your Webpack files in order to run them in parallel. I don't need to say that this will make our application faster because of a simple math's formula.

Let's say that we have two builds that are not necessarily dependent. Let's call them build A and build B. In order be more precise and didactic, the time that the application will take to finish whenever I have synchronous builds is

buildTime = A + B

And the time that the application will take to finish whenever I have asynchronous builds is

buildTime = Max(A, B)

Parallel Tasks in a Single Thread World

I won't spend time going deep into this topic because it requires a full article to explain how processes work in an operating system and how NodeJs allows us to run multiple processes.

If you want to understand better this topic, you can read the following article that gave me a very good perspective about how does Node Js handle processes.

I made a little research and I found two possible solutions for my problem:

  • Reading this thread, I found that you can return more than one Webpack configuration object, and every object will be run in a separate instance.
  • I found this package that was made by a company called Trivago. I didn't try it honestly, but if someone is interested, this is another option.

Finally, I decided to try it by myself and that is the motivation of this post.

What does this article cover

  1. Build all the javascript code into a bundle called bundle.min.js
  2. Build the sass code into a file called styles.css
  3. Use the Webpack Node Api to build the previous files
  4. Generate the stats.json file

Npm Script and Entry Point

I am using the same github repository used by my previous article, and I just added a npm script called concurrent-build to run the builds separately. The most important change is at the package.json file, because instead of using Webpack command directly, we will call a javascript file that will take care of everything.

Instead of

webpack — config ./webpack.production.config.js — progress — profile

We will have something like

node ./webpack.concurrent-build.js

Webpack Concurrent Build

In this file, I declared the pipeline needed to finish the build in which I included two configuration files

  • webpack-concurrent-app.js
  • webpack.concurrent-styles.js

The first one will output the javascript code and the second one the styles. The most important thing here is the method runAsyncBuild that goes trough this array and create a child process for each config file, and wait for a signal to kill the process.

I used the fork method for parallelising the build. It's quite simple, it just receives the path of the process that you want to run as a child_process and that's all.

Inside each configuration file, you will have a normal Webpack configuration file. The only thing that change here is the need of calling directly the compiler instance from Webpack, passing the configuration object and a callback to know whether the build finished successfully or not.

The following gist shows how the build that outputs the javascript code looks like.

The compile method is a method imported from a utils file that I created in order to be reused from multiple build files and it's the only different thing.

After calling the compile method, the program will wait for it to finish to send a message to its parent process to indicate that it has finished and now can be killed. (It's a suicide message x.x)

Webpack Node API

As mentioned implicitly multiple times before, Webpack has the ability to be called from a Node Js API.

In summary, you can get an instance of the compiler that runs inside Webpack, and call it with your configuration objects.

Taken from Webpack's public documentation

As described before, just by requiring the webpack lib and calling it with the configuration object and a callback, the compiler is called as it was called by using the webpack command from the npm scripts.

I used the default webpack method that calls the compiler if a callback is passed as a parameter.

I created the method compile just to wrap the pre and post processing of the compiler. A promise is returned in order to let the caller know when does it finish.

Inside the callback, error checks are made and if everything work as expected, the promise is fulfilled and the result is printed in the console.

Is important to say that there's a lot of things that need to be called again, like the console output with colors and statistics.

For further understanding, read the official documentation.

The styles build was made using the same concepts explained before so it’s straightforward. Feel free to look the code at github.

Generating the Stats.json file

You can find a method called generateWebpackStats inside webpack.utils.js.

The same way as the compiler method, I created it to wrap the logic related to check whether the output dir exists or not, and create it if it does not exist.

The information needed for this file is returned at the compiler's callback and passed as a parameter to this method.


Pros and Cons

You need to ask yourself (please do it!) if it's better to implement all this logic again or you can go with another approach. Not everything is beautiful and easy and it took me a good time to join all these pieces. Things like the Hot Module Replacement Middleware and console outputs need to be considered and "reimplemented" by yourself. I put reimplemented between quotes because you won't implement it, but you will have to configure and call everything by yourself. In this article, I covered the Node Webpack API for production mode only, but another article will be published considering the development mode.

Thanks for reading, and hope it helps you! :)