webpack freelancing log book (week 5–7)

2017/05/01–2017/05/21

In case you are wondering why there was no log book for the last two weeks: My daughter was born on 2017/05/05. So I was a bit busy with not-sleeping…

Anyway I had some time do concentrate on the most anticipated feature (at least according to our voting page): Scope Hoisting.

Scope Hoisting has been made popular by an awesome module bundler called Rollup.js created by Rich Harris. Rich has written a guest post in the webpack publication about the differences.

The idea is pretty simple. EcmaScript Modules (ESM) declare imports and exports in a static way, and exports/imports are live bindings, so this allows to concatenate all modules and rename variables in a smart way to fulfill the ESM spec semantics.

This has a number of benefits compared to wrapping each module in a function:

• Less functions → less runtime overhead, because less functions are invoked and less scopes are created. (Runtime Performance)
• Single scope → minimizer can do a better job, because it now knows about inter-module dependencies. (Runtime Performance)
• Smaller bundle size before gzip → parsing the bundle is faster. (Runtime Performance)

There are some disadvantages, but most of them are not really relevant, when Scope Hoisting is seen as production-only optimization.

• HMR can only replace modules if they are isolated with a function wrapper. (DX)
• Minimizing can be more expensive when the scope is bigger. (Build Performance)
• Modules can no longer processed individually, but need to be processed combined. (Build Performance)

Ok now we know Scope Hoisting is a good idea in general. Rollup already proved that it’s possible and great, let’s implement it for webpack.

But for webpack there is one additional challenge: webpack’s primary feature is Code Splitting. With Code Splitting splitting the code into multiple files, it’s no longer possible to create a single scope. Even when creating a scope per file you get into the problem that one scope need to reference variables from another scope.

So the new idea is Partial Scope Hoisting. Instead of creating a single scope, we try to find module groups in the graph which are in the same chunk and only a single module is used by modules outside of the module group (we call this module the root of the module group).

In practice such constructs are quite common. Most packages are only referenced through a single entry module (package.json main field). If we create a module group of all modules of the package and use the entry module as root, we have such a construct.

Note: While we view the world in the sense of packages, webpack doesn’t know about packages. It only knows about modules and dependencies, so it may even scope hoist between packages.

Once we found a module group like that, we apply Scope Hoisting on this group. This means all modules are concatenated, variables renamed to avoid conflicts, inner-group dependencies are resolved by using the same variable name and inter-group dependencies are kept. From now on we can thread the result as single module.

When the algorithm always finds the largest possible group and all possible groups, we reduce the number of modules to a minimum.

It’s probably good to know the factors that cause group root or prevent Scope Hoisting at all:

• non ESM → Prevent (may change in future)
• imported by non-import → Root
• imported from other chunk → Root
• imported by multiple other module groups → Root
• imported with import() → Root
• affected by ProvidePlugin or usingmodule → Prevent
• HMR accepted → Root
• using eval() → Prevent
• in multiple chunks → Prevent (may change in future)
• is entry point → Root
• export * from "cjs-module" → Prevent (may change in future)

When can I use it?

It will be an experimental feature of webpack 3. You can enable it by using the plugin:

plugins: [
  new webpack.optimize.ModuleConcatenationPlugin()
]

Until released the next webpack version is on the next branch:

"devDependencies": {
  "webpack": "webpack/webpack#next"
}

When you experiment with Scope Hoisting in webpack I would love to get some feedback. Write a blog post, gist or issue and tweet/dm me…

Now a few technical details how it’s implemented:

The whole process is composed of two parts. Each implemented in a separate file:

• ModuleConcatenationPlugin: This plugin finds module groups and creates aConcatenatedModule for each of them.
• ConcatenatedModule: This subclass of Module concatenates a bunch of inner modules and resolves inner dependencies.

The ModuleConcatenationPlugin runs in the optimization phase. The ConcatenatedModule runs in the Code Generation phase.

Grouping Algorithm

The algorithm for grouping was pretty straight-forward once I got the idea:

Idea: Start with one module, try to add dependencies, use backtracking if this fails. Recurse.

Pseudo code:

for each module M:
  create new module group MG with M as root
  for each dependency D of M
    try to add D to MG
  if modules in MG > 1:
    sort modules in MG topologically
    create ConcatenatedModule from MG and add it to chunk
    for each module in MG
      remove module from chunk
function try to add M to MG:
  if M is already in MG: return true
  if preconditions for M fail: return false
  MG' = MG
  for each module U that depends on M:
    result = try to add U to MG'
    if result is false: return false
  for each dependency D of M:
    try to add D to MG
  MG = MG'
  return true

Concatenation

Dependencies in a Module are expressed by subclasses of the Dependency class. A DependencyTemplate subclass is used to generate code for a Dependency. Usually there exist one DependencyTemplate per Dependency, but this relationship is loose coupled and there is a Map from Dependency class to DependencyTemplate so you could overwrite templates. This overwrite feature exists since the beginning but was never used. Until now…

The ConcatenatedModule uses this feature to intercept these templates (by overwriting the templates with a decorator). This way it can change the generated code for inner-group dependencies.

These steps happen when the code for a ConcatenatedModule is generated:

• Build up meta info for each contained module (exports, reexports)
• Render the original module with modified dependency templates.
  Inner-group dependencies generate placeholder variable names.
• Parse the result with acorn. Run scope analysis with escope.
• Find and rename conflicting variables.
• Find inner-group reference placeholders and replace them with name in other module.
• Generate namespace objects if used.

Other changes required

While implementing Scope Hoisting I discovered a subtle problem that occurs when writing very weird code.

// module.js
export const abc = 123;
export function magic() { return this; }
// user.js
import * as module from "module.js"
module.magic().abc

From webpack perspective abc is unused and the namespace object is only used in a static way. So webpack does a few optimization by default:

• It mangles export names. So magic is now a. abc don’t get a name because it’s unused.
• Unused export are not exported at all.

The problem: Using this in a exported function in combination with usage of module namespace objects could invalidate assumptions made for this to be correct. Scope Hoisting is affected in a similar way.

To do this 100% correct according to the spec, we would need to disable a lot optimizations. But I’ve chosen to make it incorrect in this case by default.

So before the haters start to cry: “This could break code…” There is a flag module.strictThisContextOnImports to enable the spec-conform behavior at cost of less optimized code. Feel free to use it. I guess nobody does.

So this will generate a bit less code for imports. If you’ve seen __webpack_require__.i(__WEBPACK_IMPORTED_MODULE_0__INCREMENT__["a"]](x) (minimizes to r.i(t.a)(x)) in your bundle before you will now see __WEBPACK_IMPORTED_MODULE_0__increment["a"](x) (minimizes to t.a(x)) by default.

As a rule of thumb: Don’t use this in exported functions, if you don’t .call them explicitly.

By the way we did something similar in the CommonJS implementation. The behavior for top-level exceptions in modules is not perfectly correct. Perfect behavior would require a try-finally block which has very negative impact on performance. See also output.strictModuleExceptionHandling. Did you ever notice…?

< week 4 week 8 >

webpack is not backed by a big company, unlike many other big Open Source products. The development is funded by donations. Please consider donating if you depend on webpack… (Ask your boss!)

Special thanks to these sponsors: (Top 5)

• Capital One (Bank) donated a total of $12,000
• ag-Grid (DataGrid) donated a total of $7,500
• AngularClass (Training) donated a total of $2,512
• Google Angular (Framework) donated a total of $2,500
• OpenCollective (OS funding) donated a total of $1,490
• Full list