Demystifying the JavaScript Modules Resolution

The history of JS module resolvers

MidJourney: Module resolution process

⚡Tl;dr

What: Resolve files’ module dependencies and generate an output consumable by a browser.
Why: JavaScript didn’t embed any module system as a built-in language feature, so the industry had to create one.
Next: Part 5: Bundlers, Part 6: Tooling landscape

Introduction

A little bit of history first.

The JavaScript language was designed to allow interactivity here and there inside a web page thanks to HTML <script /> tags.

So, there was no import/export system provided out of the box in the language built-ins.

But gradually, the industry saw the possibilities provided by JavaScript and started to embed more and more logic in their client-side applications.

The industry shifted to build more complex and more significant frontend applications and the need for splitting the code into many files became a necessity (for readability and maintainability reasons).

At first, the process of file linking was done manually. That’s what I call the age of pain (the OGs know).

To be able to easily handle bigger scripts, the industry created task runners.

Task Runners

(Gulp, Grunt, Make, Broccoli).

Their mission?

Automate the manual task of merging a project’s JavaScript files into a single one, ready to be imported in an index.html <script/> tag, and performing some on-the-fly code modification and/or tasks execution thanks to plugins (ie: Uglify, Unit Tests, Linting…).

Ultimately, task runners generate a JS file that includes your whole application in the form of a concatenated set of IIFE (CF: the below diagram).

Task Runner flow

This trick aims at limiting the pollution of the global scope. Since no module system was implemented, the plan was to use the various level of scopes available in the language (Block scope, Function scope, Global scope).

Task runners were the first step toward bundling automation but still had annoying flaws.

Another big issue was the necessity to manually respect the file declaration order (lack of dependency resolution).

Module import order

Here, fileOne is imported by fileTwo.
And, main imports both fileOne and fileTwo.
This means that their declaration order must look like the following:

<script type="text/javascript" src="./fileOne.js"></script>
<script type="text/javascript" src="./fileTwo.js"></script>
<script type="text/javascript" src="./main.js"></script>

Any other order will result in a definition error: code using code that is undefined because it hasn’t already been declared.

To get this annoying issue out of the way, JavaScript needed a module resolution feature. And long story short, lots of parallel initiatives started to pop out to fit that role by either providing a standard and/or proposing an implementation of that core missing feature.

Module resolvers

Throughout the last 15-ish years, “a few” systems have been created to fix the particular issues of module resolution and loading.

Here’s what it looks like:

History of module resolution syntaxes & loaders

Each library aims at fixing a particular issue. Incrementally combining or merging them allowed the industry to slowly lean toward a way to dynamically and asynchronously resolve and load files inside a web browser.

Let’s walk through the major evolutions, highlight what they fixed and how they can be categorized.

IIFE
The age of pain.

During that age, JavaScript applications were wrapped in Immediately Invoked Function Expression.

This was painful and not ideal for various reasons, at the top of my head I’d say maintainability, readability, scaling, collision issues, and accidental synergies among scripts that might be instantiated into the global scope.

Here is how it was done:

(
  function(){ /* code injected in the main scope */ }
)();

CommonJS (2010)

Then, came CommonJS.

CommonJS is not a module resolution system, it’s a consortium (like the TC39). It can even be considered a project whose objective was to create a standardized syntax convention for the module ecosystem.

This syntax has been implemented for backend applications and heavily relies on its access to built-in internals like OS features (i.e manipulation of the file system) which made it synchronous by nature.

The project gave birth to the following syntax: (there’s a lot of chance that you might be familiar with it because it’s the one used by older versions of NodeJS).

// myFunction.js declaration
module.exports = function myFunction(args){
  // do stuff...
}

// myFunction.js import
var myFunction = require('myFunction.js');

Browserify
Documentation

Meanwhile, module bundlers (more on the bundlers in the next post of this series) like Browserify were created to close the gap between server and client-side systems.

Indeed, CommonJS (designed for server-side usage) is synchronous and thus not ideal within web browsers. Blocking the main thread with synchronous processes is what we politely call a bad practice.

Browserify came in to allow client-side asynchronous module resolution and aimed at stitching the modules together at build time.

Usage example:

// file.js
var titi = require('./file2.js')
const a = 'toto' + titi

// file2.js
module.exports = function a(){
    return 'titi';
};

$> npx browserify file.js -o bundle.js

// bundle.js
(function () { function r(e, n, t) { function o(i, f) { if (!n[i]) { if (!e[i]) { var c = "function" == typeof require && require; if (!f && c) return c(i, !0); if (u) return u(i, !0); var a = new Error("Cannot find module '" + i + "'"); throw a.code = "MODULE_NOT_FOUND", a } var p = n[i] = { exports: {} }; e[i][0].call(p.exports, function (r) { var n = e[i][1][r]; return o(n || r) }, p, p.exports, r, e, n, t) } return n[i].exports } for (var u = "function" == typeof require && require, i = 0; i < t.length; i++)o(t[i]); return o } return r })()({
  1: [function (require, module, exports) {
    var titi = require('./file2.js')
    const a = 'toto' + titi

  }, { "./file2.js": 2 }], 2: [function (require, module, exports) {
    module.exports = function a() {
      return 'titi';
    };

  }, {}]
}, {}, [1]);

Asynchronous Module Definition (AMD)
Documentation

The synchronicity nature of the existing solutions was an issue for client-side applications, so a new module resolution syntax standard has been created: AMD. It literally means Asynchronous Module Definition.

The AMD specification defines a single function called “define” with the following signature:

define(
  id?, // specifies the id of the module being defined
  dependencies?, // array literal of the module ids that are dependencies required by the module that is being defined
  factory // function that should be executed to instantiate the module or an object
);

This API takes its roots in the CommonJS initiative but finally drifted apart as the sub-groups representing these two standards couldn’t find any consensus.

A version of this API started on the CommonJS wiki as a transport format, as Modules Transport/C, but it changed over time to also include a module definition API. Consensus was not reached on the CommonJS list about recommending this API as a module definition API. The API was transferred over to its own wiki and discussion group.
- AMD documentation

At this point, AMD provided a way to avoid global scope pollution and stressful moments keeping the dependencies declaration order right, but it was “just” the definition of the standard and no implementation using it existed at that time.

Example using the AMD API:

// Module definition
define(
    //The name of this module
    "some/script",

    //The array of dependencies
    ["dep1/dep2"],

    //The function to execute when all dependencies have loaded. The
    //arguments to this function are the array of dependencies mentioned
    //above.
    function (dep1) {
        function dep2 () {
            // ...
        }

        //This will now work
        dep2.prototype = new dep1();

        //return the Manager constructor function so it can be used by
        //other modules.
        return dep2;
    }
);

// Module Import
require(["some/script.js"], function() {
    //This function is called after some/script.js has loaded.
});

RequireJS
Documentation

Thus, RequireJS proposed an implementation of AMD’s asynchronous concept and syntax specifications. It can be considered a module loader that is executed at runtime.

Here is the full Architecture Design Record which explains what led the team behind RequireJS to choose this solution.

Its usage is pretty simple, everything happens through an HTML <script /> tag. When parsed and executed by the browser, the script loads the require.js module inside the current runtime which will in its turn starts an asynchronous module resolution starting with the file given as data-main attribute (main-script.js in the following example).

Usage:

<script src="require.js" data-main="main-script" />

Universal Module Definition (UMD)
Documentation

Later on, supersets like UMD tried to homogenize the usage of the existing module resolution systems and syntaxes by providing a universal abstraction. It infers at runtime which module resolution syntax is available in the current browser’s execution environment and uses the first one found.

It’s merely an if-else expression that checks if the current running environment implements tooling using one of the two main syntaxes: AMD or CommonJS.

Example using UMD:

(function (global, factory) {
    if (typeof define === 'function' && define.amd) {
        // AMD
        define(
          ['dep1', 'dep2'],
          factory
        );
    } else if (typeof exports === 'object') {
        // Node, CommonJS-like
        module.exports = factory(
          require('dep1'),
          require('dep2')
        );
    } else {
        // Default global scope
        global.newGlobalFunction = factory(global.dep1, global.dep2);
    }
}(this, function (dep1, dep2) {
    // Code that uses dep1 and dep2
    // ...
}));

ES6 modules
Documentation

If you began to feel lost in all these module resolution syntaxes and loaders, I have good news for you: you might not need to know them all for your future projects thanks to the ES6 modules.

ES module provides specifications for module resolution at the heart of the JavaScript language, which makes all the above solutions obsolete workarounds.

The ECMAScript’s specification is available since 2015 and has since then been implemented in almost every major web browser.

CanIUse: ES Modules implementation in the browsers market

Syntax Example:

// toto.js
export const toto = 'toto'

// titi.js
import { toto } from 'toto.js'

const titiOne = 'titiOne'
const titiTwo = 'titiTwo'

export { titiOne, titiTwo, toto }

// main.js
import { titiOne, titiTwo, toto } from 'titi.js'

// Use toto, titiOne and titiTwo

<!-- Load the module from the HTML page -->
<script type="module" src="main.js"></script>

This code doesn’t need any transpilation since it’s understood natively by every latest browser. That being said, if your use case requires your application to be used on older browsers, you won’t cut it, but at least your code will be “future-proof”.

SystemJS
Documentation

And last but not least, the ultimate evolution of module loaders is SystemJS.

It’s able to load modules at runtime written in any syntax (CommonJS, AMD, UMD, ES6 Modules) and is built upon the ES6 module loader polyfill.
It can also be configured (and enhanced through plugins) to perform certain tasks on the fly like transpiling or compiling the loaded source code from various origins (TypeScript, CoffeeScript, and any ES syntaxes) into JavaScript of a given target ES version.

It can be used either through an HTML script entry point:

<script src="system.js"></script>
<script type="systemjs-module" src="/js/main.js"></script>
<script type="systemjs-module" src="import:name-of-module"></script>

Or through dynamic imports within your application:

System.import('/js/main.js');

Note:
SystemJS is a perfect candidate to stitch up Microfrontends since it supports almost any syntax and is executed at runtime. It is by the way recommended by Single-Spa as the go-to solution for module resolution when using their Microfrontend router library.

Conclusion

This concludes our journey through the evolution of task runners, module resolvers, module loaders, and their different syntaxes.
15 years of upgrades, overlaps, and replacements lead us to a pretty big ecosystem where lots of solutions coexist within an extensive ocean of tooling.

In the next part of this series, we’ll see how the industry solved the remaining JS challenge of bundling an entire application. By exploring the bundlers’ possibilities and trying to highlight the major players’ main features.

Next up, Part 5: Bundlers

Thanks for reading!
👏🏻 Give me a clap and “follow” if you enjoyed this series.