It’s a wrap: Google 🌞 of Code
After more than 52 merged PRs and 75 commits in webpack’s CLI, here we are at the end of the Google Summer of Code journey.
Webpack CLI encapsulates all code related to CLI handling. It captures options and sends them to webpack compiler. You can also find functionality for initializing a project and migrating between versions.
Today, we’ll share about what all happened in this little journey of webpack-cli. We’ll majorly talk about the overall work that happened during this phase and also about our finding from the learnings and experiences we had.
Abstract Syntax Tree Optimisations 🌲
Abstract Syntax Tree or AST is a hierarchical tree-like representation of the code. It helps us understand and read relationships between different elements of the code. An AST transformation is an action of modifying this tree (i.e. changing an existing AST to a new AST).
Let’s take a look at how an AST looks for a simple webpack config like this:
Each and every part of our code is formed of AST nodes which can be of various types like Object Expression, Member Expression, Literals, Variable Declaration, etc.
Once we have an AST of the full webpack config, we can manipulate or transform it for implementing useful features like migrating webpack configuration for version upgrades or also to add, update or remove properties from existing configurations.
For doing all the manipulations and transforms in AST, we use a library called jscodeshift developed by Facebook. This is the same library which is used by React folks for developing react-codemod. jscodeshift provides a nice interface for playing around with AST paths, node builders and traversals.
We have a lots of utility methods written around the jscodeshift API which we use as wrappers in our commands. Earlier, we started with using a non-recursive approach towards AST transformations for generation of new AST nodes in the init command. As new features and cases came in and the nodes scaled, we realised that this was not an efficient way of handling transforms as each case needed to be handled differently depending on the kind of node we want to generate and attach to the tree. It was hard to maintain deep nesting for nodes as well. This resulted in a lot of code and utils which were hard to scale and maintain.
To fix this, we took a recursive approach for adding and updating AST nodes which handles all types of nodes and works with deep nesting implicitly. Hundreds of lines of code reduced to 50 lines recursive function. It was a huge maintenance improvement for team and potential to implement new commands like add/update. Kudos to Even Stensberg!
This enabled implementing add or update command which is one of the newest command we shipped a while ago that enables users to add new webpack properties from the CLI directly. We merged the logic used for creating and updating of AST nodes to support this new command.
Remove command is another newly released commands which is used for removing existing properties from webpack configuration files. Our end goal for this command was to remove a specific property from the config file without disturbing the rest of the contents and validating it.
One of the initial thoughts we had were to somehow reuse the recursive function used for adding so we thought of re-generating a new AST without the node selected to remove. This was costly due to regeneration of the entire tree. In fact, we didn’t need any recursive approach for removal of AST nodes as we only have to remove a targeted node leaving rest of the tree unaffected. For instance, in loaders/rules, we don’t need to drill recursively inside each option for each rule. We show a list of rules based on their names (picking up their loader property) and then a prompt to remove only a specific targeted rule object from the rules array. For objects like devServer, we show a list of keys like port, contentBase, filename, etc. It made more sense to follow the non-recursive approach for this use-case.
TypeScript migration
Migration from JavaScript to TypeScript was one of the most discussed open issue for more than 6 months.
Introducing TypeScript gave a good stability to the codebase and help us trim down documentation at some places. It also unlocked code completion, prediction and reduced the likelihood of runtime errors and help form a good internal code documentation. It’s a big win for all.
We started with a gradual migration with help of allowJs flag so that we can migrate the codebase in chunks.
TypeScript with Monorepos 📦
Recently, we migrated to a monorepo strategy using lerna which allows us to ship small mono packages instead of a big monolith package. To follow the principles of mono packages, we integrated TypeScript build system not just at root level but also at every package level since each package should be independent of each other. We also setup linting using TSLint following our previous ESLint style.
One of the challenges we found while working with TypeScript is a little initial cost for setting up all the custom interfaces for all the methods and objects we’re using in the code. It’s definitely worthwhile of an exercise. To add typing support for npm modules, we straightaway went with the DefinitelyTyped type definitions. Also, some of the modules like chalk come pre-bundled with types. For npm modules that neither come with types nor have DefinitelyTyped support, we created custom module interfaces again. Integrating custom module type definition is easy with help of Triple-Slash Directive that uses a reference path but in most cases, the TS compiler is smart enough to pick all the declaration files. Using custom module types made us choose the signature and shape of each method and object exposed by the module. We needed to be cautious enough not to mismatch the underlying JS behaviour used by the libraries.
We also plan to add the missing modules to DefinitelyTyped by contributing the declaration files we created for the npm modules missing types. We haven’t done that already since we only wrote types for the method we were consuming in our codebase.
TypeScript compiled code is not always backwards compatible ♻️
The core entrypoint of the CLI is a bin folder which contains logic for handling of all the flags and command options. We decided to stick with JavaScript for this part of the codebase in order to keep friction low for new contributors who might not be familiar with TypeScript. We’re still evaluating how new contributions are impacting after this big change in language, style of coding and might shift to TypeScript for all files once we’re confident that everyone is comfortable with it.
In order to stitch TypeScript compiled code with this existing JavaScript code, we had to make some minor with the way we are importing this compiled code. One particular thing we would recommend is avoiding default exports in TypeScript if you want to import your compiled JavaScript code into other parts of the codebase using JavaScript.
To understand one of the pain points of working with TypeScript and JavaScript code together, let’s take a look at how the TypeScript compiles code to JavaScript:
Notice how the compiler created a named export with variable default instead of module.exports. Now, in order to import this compiled module in JavaScript, we have to use a require syntax like this:
Migrate Configs - Webpack 3 to 4 🔢
webpack 4 (Legato) was released around 5 months back and is one of the biggest releases webpack has done.
Most new use-cases worked out of the box with the new #0CJS support and sensible defaults. A lot of changes were made in the existing API to unlock higher optimisation levels. One of the biggest change in the public API is the transition from CommonChunkPlugin to SplitChunksPlugin.
Originally, chunks (and modules imported inside them) were connected by a parent-child relationship in the internal webpack graph. The
CommonsChunkPluginwas used to avoid duplicated dependencies across them, but further optimizations were not possibleSince webpack v4, the
CommonsChunkPluginwas removed in favour ofoptimization.splitChunks.
We want to make this transition smooth for our existing users using the nifty migrate feature we already had for migrating from version 1 to 2. We also show a nice little diff on what exact changes the CLI is going to do in the config during migration process.
We started with version 3 to 4 migration during mid-April and are happy to announce that we’re getting the final pieces to get in place for a smoother transition. We will release this big feature soon after some internal testing.
Some of the PRs that went into this work: #558
Miscellaneous improvements
- Added autocomplete support for configuration properties for
addcommand. #502
- Upgraded major version for all dependencies. #290
- Greenkeeper support for automating dependency management.
- Support for custom paths to webpack configuration files for
addcommand. #501
Lessons Learned 💂
Working on big open-source projects and tough and exciting. The best part is having your work directly impacting hundreds of thousands of developers who use webpack.
There are a lot of learnings that comes along with experience and familiarity with the core team and community. Collaboration and team effort are key for the success to any large scale project.
Community matters a lot. We should never forget the end goal of what we’re trying to solve and always listen to what the community needs and the concerns they share.
Adapting to existing code and following design patterns help maintain consistency throughout the application. Always write composable, maintainable and loosely coupled code which makes scaling easy.
GSoC and webpack is probably the biggest breakthrough of my career. Can’t thank enough to Google and webpack for giving me this amazing platform. This opportunity definitely has a very huge impact on me. It has allowed me to grow from an open-source contributor to an open-source maintainer.
Big shoutout to
, and for helping me throughout the programme and being such a good mentors and teachers. ❤️
Next on the CLI roadmap 🚀
Some of the things that we have next on the roadmap are refactoring the entrypoint folder to decrease some bloat and making it easier to maintain, improvements in the documentation along with the release of new commands and lots of user-experience revamp like an introduction of an interactive mode or dashboard.
