Sharing third party dependencies in a micro-frontend landscape using Webpack

A new strategy to improve the performance of the ASOS website

A typical need with micro-frontends

While it’s good practice for teams and micro-frontends to be as independent as possible, there are some things they should have in common e.g. component libraries.

And what about the external third party libraries which each require micro-frontends to run? Improving third party performance definitely occupies a big space in our boards — it goes without saying that we all would like our apps to produce smaller bundles.

On the front-end side, one of the strategies we currently adopt involves a fairly recently implemented tool with the purpose of producing a JS bundle, `dll.vendor.xxx.js`, a JS bundle containing versions of some hand picked third party dependencies that more or less are used in every micro-frontend we maintain: react, react-dom, redux, classnames are to name but a few.

Each micro-frontend then has some webpack configuration to exclude the bundling of those specific dependencies at build time.
All that is left to serve is the `dll.vendor.xxx.js`bundle on each page. It’s a sensible approach and it works, each micro-frontend is basically saving this much space (brotli compressed):

We call that tool ‘SharedDeps’.

We can do better

Dealing with SharedDeps made me realise that the solution, while essentially doing its job, in practice, suffered from these issues:

1. Rolling out breaking changes in the third party libraries controlled by SharedDeps implies mobilising all the teams to adapt their projects to the changes.
2. If a consumer of SharedDeps doesn’t need part of it, a cold user will have to download unnecessary code at that stage.
3. The tool adds complexity. It’s an extra layer, with its own pipeline, it needs to be deployed and maintained.

There must be another way.

The idea

Webpack 5 and its new exciting Module Federation feature were yet to be proposed, Zack Jackson was just starting to share his brilliant ideas about front-end architectures. Furthermore, upgrading all the apps to bundle our code with webpack 5, which is still only in beta, will not be happening any time soon unfortunately.

So I started jotting down ideas on the whiteboard and realised the following:

Given two or more distinct micro-frontends with different webpack configurations, assuming they both import the same version of the same dependency, eg. react 16.10.1, there should be a way to set webpack to produce the exact same chunk for that dependency (eg. react.16.1.1.js). This should be regardless of the configuration and the environment in which webpack is being ran — literally the same filename and content.

If that’s possible, then all we need is to have those javascript chunks uploaded in a space every consumer can have access to.

Apps would ‘pick’ their special third party library chunks from this shared address as needed. But some apps would have produced the same third party library chunk, so effectively they would request the same file in the same path. The first app needs a download, the second has it cached.

You see where I am going with this? Let’s assume we have that solution already coded, I can imagine the following benefits:

• Increased performance in a micro-frontend landscape by sharing dependencies more efficiently. Avoiding duplicate dependencies equals smaller bundles and already-cached dependencies throughout the site.
• No more centralised controlling tool, no extra deploying needed. The micro-frontends are therefore more isolated.
• Avoid mobilising the entire web department when single apps decide to upgrade dependencies. Allow breaking changes to be rolled out without lock-step.
• Bump individual deps without re-downloading the entire vendor bundle.
• Tracking and Control — easy to take actions and added consistency between multiple versions of the same third party library across the estate.

So, did we make it then?

Indeed we did.

All I can say is that I’m honoured to have had the support I had in ASOS while I was playing with this idea.

It’s in the ABC of ASOS:

Authentic
Brave
Creative

It’s been a complex journey, from that intuition to the final form. I personally made a few mistakes but I learned a lot. I’ll share my learnings at the end of the story but let’s just say teamwork is what kept the project alive and made it possible to reach production.

Introducing SharedDeps v2

I made this quick video to introduce you to the solution:

https://vimeo.com/manage/videos/452542328

And now a more in-depth look

It’s probably best to be up to date with the way webpack operates, especially the importance of its plugin system: Plugins, concepts, Writing a plugin.

Final form

We went through a lot of code refactors but in the end we managed to pack everything into a simple webpack plugin.

This is how you implement it in a consumer app — pretty straightforward:

webpack.config.js

The nine modules

This is what the entry point of the plugin looks like:

These nine modules, similar to plugins, make use of the compiler and compilation hooks to manipulate the bundle creation.

I’ll try to sum up the role of each module, in order of first appearance in the compilation process:

1) addMetaData module

Firstly, it finds and stores data coming from the package.json of each third party library the App requires, to be used in other modules.

Essentially it stores dependencies lists, versions, semver versions. Info.

2) asyncChunks module

When using dynamically loaded modules, calls to import() are treated as split points, meaning the requested module and its children are split out into a separate chunk. When something coming from a node_module is used exclusively in one of the children of such modules, webpack bundles the third party code with the rest of the chunk code.

This module temporarily moves the ownership of every third party module imported in an async chunk to the closest initial parent chunk, allowing SharedDeps to process them.

3) deOptimizeModules module

It performs actions to prevent webpack from applying certain optimisations, such as flagging modules to be then optionally optimised by loaders or plugins, it also prevents dead-code-elimination in the shared modules therefore producing the same chunk content (and hash) regardless of its use.

Example: react-dom exports certain functions, such as render or hydrate.

This module makes sure that both functions are included and chunked.

4) setModuleIds module

Used to set the ID of each module contained in each chunk.

Each module ID is generated by hashing its own source code. This guarantees deterministic hashes as a result.

5) makeChunks module

This module simply configures the webpack SplitChunksPlugin which effectively does most of the heavy lifting in the SharedDeps plugin. It’s the webpack plugin that does the actual code splitting.

The code for this module looks like this:

6) setChunkIds module

It sets the ID of each shared chunk by iterating over each module that particular chunk includes, creating a hash based on the module IDs (which we previously produced as deterministic hashes in the setModuleIds module) that chunks contain.

7) setJSONP module

https://webpack.js.org/configuration/output/#outputjsonpfunction

It makes the JSONP function name unique to the consumer by appending a unique hash to the app jsonpFunction webpack config. It also creates and handles an extra JSONP function specifically made to store and load shared-deps chunks.

It solves the following problem:

If multiple webpack runtimes (from different compilations) are used on the same webpage, there is a risk of conflicts of on-demand chunks in the global namespace.

8) mirrorAssets module

This module is in charge of storing the source code of each chunk, minifying it with our bespoke rules and bypassing any tersing option set in the consumer. This ensures the consistency of the code: consumers may have different output optimisations that should not impact our ‘protected’ shared deps chunks.

9) makeManifest module

This last module produces a manifest containing the array of shared chunks produced by a consumer, eg:

This is not a core module of SharedDeps as you may have imagined but we’re quite excited about its potential.

The aggregated data coming from all the SharedDeps manifests can be used for monitoring and tracking third party dependencies across the whole web estate. Specifically, we are now able to pinpoint any dependency discrepancy and produce actions to save us space.

Imagine two front-ends require the same library but not the exact same version:

App A: library 1.0.2 (35kB compressed)

App B: library 1.0.3 (25kB compressed)

There’s an optimisation there! Let’s update App A and save our users 35 kb!

Here’s a monitoring tool that aggregates manifests coming from all the front-ends implementing SharedDeps:

With this kind of information we can definitely make improvements in our codebases:

Here we have two apps that could potentially save space by upgrading their version ofjs-cookie to the most up-to-date version currently consumed.

Complete flow of shared deps and webpack hooks

The following diagram illustrates, in firing order, how SharedDeps utilise webpack hooks.

Et voila’. Head over to your ‘My Account’ page on the ASOS website to see SharedDeps in action.

The ‘implicit’ sharing of dependencies

Let’s say a user lands on MyAccount, one of our micro-frontends.

My Account, network tab, chrome developer tools

It’s the first app the user hits, so they need to download each script.

Let’s now say they move to another app, the one that manages the newsletter subscriptions in ASOS for example:

See that? We didn’t need to download the majority of those scripts.

Interesting, right?

Performance

While tests on our apps in production are still ongoing, you can have a look at this lab test using two controlled apps, App1 and App2 (imagine two micro-frontends that don’t talk to each other), loading an amount of common dependencies and some unique ones. Two sets, with and without shared-deps.

Imagine a user landing on App1 and moving on to App2.

green line FTW :)

The test confirmed what we expected:

Shared-deps belong to the group of long-term-caching optimisation strategies, so unfortunately cold users may face a small hit.

There are two main reasons.

By design, shared-deps need to prevent dead code elimination when producing the third party chunks. Otherwise, two apps requiring two distinct methods from a library would end up producing two chunks of the same third party library, possibly the same version but with a different hash in the filename.

For the second reason, code minification is less efficient when the code is split into more chunks.

Let’s look at the bright side now

With shared-deps, when the user moves from App1 to App2…did you see that green line in the SpeedCurve (great tool) screenshot?

That’s the initial SharedDeps idea right there, dozens and dozens of coding hours later. There it is, recorded for the posterity in SpeedCurve (amazing tool).

Add now App3, App4, App5…there is a performance gain there, we believe.

But like we said, there is also a trade-off to consider.

In this case, the user saves 43 kb on their journey throughout the apps, but the initial amount of JS to download increases, slightly.

Personal learnings

It took months to get me from the first proof of concept to the first deploy in production. I wish I could say it went smoothly but the truth is that mistakes were made. Here is one lesson I learnt.

It’s essential to start involving people early in the process, especially in planning or if the project requires concerted cross-team effort. I definitely kept this as a sort of personal side project for far too long and therefore failed to involve the key people I needed to transform a pet-project into a cross-team tool in a reasonable timeframe. Next time, I’ll pause the coding slightly earlier and start working on the roll-out plans with those who should be involved, before going back to the Code Editor.

So, what do you think?

We are currently in the process of rolling out SharedDeps and I’d be delighted to share our findings. Do you think it has a future on the ASOS website? Do you want to know more?

Would you like to see the code?

Let us know with a comment.

Thank you for reading.

My name is Albino Tonnina and I’m a senior software engineer at ASOS. I work in the Web department.