GSoC 2019 Dev Server Refactor: WebSockets, ws, and CLI
This Google Summer of Code, I worked on a couple of key components in the webpack Dev Server: the client-server WebSocket communications, and the CLI. Most of my work was focused towards PRs for webpack-dev-server, with occasional, project-related work in webpack-cli and documentation work in webpack.js.org.
The Dev Server is a tool that allows users to develop and serve webpack apps with live reloading browser capabilities. I’ll be sharing what I have added and changed, how my efforts will impact this tool, and what lies ahead.
Brief Overview of Changes
The major goals of my GSoC with the Dev Server were completed. These goals were: adding ws and native WebSockets as a transport option to eventually replace SockJS, refactoring the CLI, and improving tests. Here is the status of these changes, and links to the relevant work:
- transportMode option, allowing users to specify ‘ws’ in place of ‘sockjs’ (merged and published, relevant PRs)
- CLI Refactor, minimalizing CLI/API inconsistency (some published, others merged and awaiting next major release, relevant PRs)
- wepack-cli serve refactor (merged and awaiting next major webpack-cli release, relevant PRs)
- Test improvements, including end-to-end tests and adding tests for untested features (merged and published, relevant PRs)
- Documentation of my features and changes (some published, other docs awaiting major Dev Server release, relevant PRs)
Luckily, virtually none of my work went to waste or went completely unused. There is still work to be done in refactoring the Dev Server, but everything directly related to my major project goals was completed.
Native WebSockets and ws
The first goal of my project was to make ws the new default server, a WebSocket server which can communicate with native browser WebSockets. These new defaults are meant to replace the current SockJS client/server implementation that the Dev Server uses. The motivation behind this change is that SockJS is poorly maintained and slower than ws with WebSockets.
Backwards and Browser Compatibility
The difficulty of simply scrapping SockJS is that it offers support for older browser versions that do not support native WebSockets. My solution had to make it easy for users to swap back to SockJS, if they needed to support older browser versions for their projects.
My new system also had to do everything that the current SockJS server and client were already doing, preventing any breaking changes, aside from the browser compatibility issue. This meant that ws and native WebSockets would have to seamlessly swap with SockJS as the client/server implementation.
WebSocket-like Interface
In order to accomplish my goals, I first built a WebSocket-like interface for both the Dev Server client and server. To get the most out of this interface, I made it almost completely unassuming of what client/server library is being used.
The chosen server library simply needs to listen for new connections, produce a connection object for each new connection, and be able to send messages to this connection object, among a few other things. In essence, the server interface looks like this:
Then, using ws and slightly simplifying, the onConnection method implementation might look something like:
The client interface is similar, requiring simply that the client library can connect to a server and listen for messages.
The reason for this design is that ws, SockJS, and native WebSockets could then all be implemented via this unifying interface. This also has the added benefit that other WebSocket client/server libraries can be implemented in place of SockJS or ws.
devServer.transportMode option
Using this interface, I implemented two modes: ‘sockjs’ and ‘ws’. Then, I added a new transportMode option, with which users can, for example, do:
Using this mode makes use of the ws server and native browser WebSockets.
Custom, user-implemented Client/Server
Beyond the simple ‘sockjs’ and ‘ws’ modes, users can now implement their own transportMode using the interface mentioned above. Instead of specifying an already implemented mode as a string, you can provide:
These are absolute paths to files exporting classes that implement the interface mentioned above. As I said earlier, this is simply a customizable benefit that came from creating a versatile, unassuming, WebSocket-like interface.
But what can you actually do with this customizability? The main use case would be to implement a specific server/client library that you desire, rather than using the provided sockjs or ws options. The key is that the client and server must be compatible to communicate. Let’s say, for example, that you do not want to use ws as a Node WebSocket server, but instead want to use websocket, another Node WebSocket server implementation. You still want native browser WebSockets. First, you would use the server implementation interface and plug in the websocket library:
To make use of this custom server implementation, paired with the native browser WebSocket implementation that is provided, you would simply do:
With these tools now available, users will be more capable of customizing the Dev Server to their needs, and the Dev Server will become easier to maintain as standards and transport libraries change.
CLI Refactor
The next step of my Google Summer of Code was to start refactoring the Dev Server CLI. The motivations behind this refactor were:
- There is inconsistency between the Dev Server API and CLI usage, making some options confusing to use
- Moving the CLI into the serve package of webpack-cli will improve the user experience, as webpack-cli is focused on making it easier to create and modify webpack projects
Removing CLI config changes
Inconsistency between the Dev Server API and CLI stems from the fact that the CLI utilizes the API, but first makes some configuration changes to the user’s provided CLI options. My thought process was that, if these changes are made to make the user experience better, API users should receive the same changes, and therefore the same benefits and consistency.
Let’s look at a few lines of code directly from the Dev Server to demonstrate what I mean. These lines of code are only executed when the user is using the CLI:
options is the options object that the user has provided in their webpack config file, and firstWpOpt is the webpack options object that will be passed into the compiler, also from the user’s webpack config file. This makes the user experience better because it infers the user’s desired watchOptions from the webpack config they provided, meaning the user does not have to include these options twice for both the webpack compiler and the webpack Dev Server.
API users did not get this feature. This is one of many examples where the API did not make changes to the options consistent with the CLI.
My solution is simple to explain: I started moving all configuration changes from the CLI to the API. Ultimately, I planned to design the CLI such that it simply parsed the CLI arguments, then passed that parsed argument object directly into the API, with zero configuration changes. In a very boiled down example, the CLI should do the following:
Breaking Changes
Because I wanted to bring the CLI and API inconsistencies to a minimum, there were many minor breaking changes involved. The example earlier of inferring a Dev Server option from the webpack options, for instance, would be a minor breaking change for the API.
For this reason, all of my changes to the CLI will be coming in the next major Dev Server version. Ultimately, these breaking changes will have a positive impact.
webpack-cli serve
My work on this command in the serve package of webpack-cli is part of the next major version effort for webpack-cli. Previously, the webpack-cli serve command simply made use of the Dev Server CLI. My goal was to make this command instead use the Dev Server API directly.
Since webpack-cli is also undergoing a refactor for the next major version, my work here involved bootstrapping the serve command using the new webpack-cli infrastructure, and parsing the Dev Server arguments using command-line-args, rather than the previously used yargs.
From these CLI improvements, users get many benefits: more consistent and predictable usage between the CLI and the API, as well as a better user experience from webpack-cli.
Dev Server Tests
While working on my main feature and refactor efforts mentioned above, I also dedicated some time to improving test coverage and stability. My main work here involved:
- Adding end-to-end tests and improving their stability
- Adding tests for previously untested features that I dealt with
What’s next for the Dev Server
My goals for GSoC 2019 were accomplished. I implemented ws and native WebSockets via the transportMode option, and I refactored the Dev Server CLI. Some of these changes have been released, and some have yet to be released as of the writing of this post, as they will arrive in the next major versions of webpack-dev-server and webpack-cli. But there are still things to be done to improve the Dev Server, which I hope I can continue to be a part of!
Upcoming for the webpack-dev-server will be the version 4 release, as the final version 3 release has already been published. This will have to coincide with the next major version of webpack-cli, as webpack-cli serve changes will rely on major changes in webpack-dev-server. Other things to expect for the Dev Server in the future will include a refactor to organize the client directories more thoughtfully, and possibly TypeScript, to name a couple.
This Google Summer of Code has been a great experience for me, and I’m very thankful that I got to do my work for webpack! I plan to continue contributing to webpack and hopefully expanding beyond the Dev Server.
