Module Loaders: Master the Pipeline!
This article is for developers who want to dig into JavaScript Module Loaders. We will look at how module loaders work, what the stages of the pipeline are, and how they could be customized.
If you are new to modules in JavaScript I would recommend to start with this article by Addy Osmani.
Intro
The nature of a human brain is that it cannot deal with a lot of objects at the same time (Miller’s Law). If you are going to build a large JavaScript application, you should stop, remember this limitation and start thinking in terms of modules.
Modules are a way to organize your applications. Simply break your functionality into small pieces thinking about how they will work with each other, and then assembly them together. A module could be seen as a black box with a clear and preferably simple API. It is common that modules can depend on other modules.
Before we continue, let me ask you a question: if you have an ES6 application, is it possible to use a module written in CoffeeScript or a CommonJS module?
If you answered “htob, seY”, you got a point! How is this possible? Module Loaders are here to help us!
Table of Contents:
1. Module Loaders
For modern web development, the following module standards are available:
- AMD — Asynchronous Module Definition, good for loading modules asynchronously (dynamic import).
- CommonJS is widely known for being used in NodeJS. It is good for synchronous module loading (static import) which works well for server-side scripting.
- ES6 — WHATWG’s module standard, is still a draft, will become the official standard for JavaScript modules. It allows both static and dynamic imports.
They have different but similar APIs and serve the following tasks:
- define a module (module syntax);
- load a module.
Note: In this article we will focus on the second task of how a module can be loaded, and get a gist of what all module loaders do.
A module system aims to simplify your development: you can focus on your current module and only have to care about what modules you directly depend on. The module loader does all the heavy lifting: performs the loading task, acts as a dependency manager and maintains a Module Registry (an object that keeps track of all modules and stores theirs source code along with other meta data).
Let’s look at how WHATWG specification describes what a module loader should do:
The JavaScript Loader allows host environments, like Node.js and browsers, to fetch and load modules on demand. It provides a hookable pipeline, to allow front-end packaging solutions like Browserify, WebPack and jspm to hook into the loading process.
Loader is a system for loading and executing modules, and there is a way to participate in the process. There are several Loader hooks which are called at various points in the process of loading a module. The default hooks are implemented on the Loader.prototype, and thus could be overridden/extended.
2. Loader Pipeline
In the diagram you can see the different stages that the Loader passes through:
Note: WHATWG (ES6) module standard defines four stages: “Resolve” replaces “Normalize” and “Locate”.
Normalize Phase
During the Normalize phase the Loader converts the provided name into a Module Identifier that will be used as a key to store the module’s data in Module Registry. The given name could be a relative path to the resource, it also could contain a shorthand mapping to a certain path, or any other logic that a particular Loader implementation provides.
Locate Phase
The Locate phase serves to determine the final resource address that the Loader will use to fetch the resource from. It is either a URL (if the host is the browser), or a path (if the host is a NodeJS server).
Fetch Phase
During the Fetch phase Loader fetches the resource by provided address. It could be that module’s body is provided to the Loader directly, in which case this phase will be skipped. The result of this phase is a string with the source code of the module.
Translate Phase
The Translate phase is probably the most interesting, because pure JavaScript is not the only way to program for web. There are a lot of popular options: TypeScript, CoffeeScript (with all its dialects), Elm, Flow, next generation JS standards, etc. Technically, there is no limit for what could be used. You can use any programming language if you can provide a JS translator that will compile your code into JavaScript.
Instantiate Phase
During the Instantiate phase module’s dependencies are loaded and linked together, then the module gets evaluated.
3. Loading hooks
Now let’s see how the process could be customized. For each of the stages there is a hook, which is a method that will be called with certain arguments. A hook can either return an immediate result or a promise.
When you override the loader’s hook method you can also call the original method. In this case you will have to pass it the parameters as defined by the hook’s signature. Alternatively, you can just return the expected result.
For an example we will look at how module my.js imports module math.js. Both are stored in the same folder “utils” (look here for ES6 module syntax):
Normalize: (name, referrerName, referrerAddress) → normalizedModuleName
Module Loader calls this hook passing three arguments: name, referrerName (the normalized name of the module that initiated the import), referrerAddress. The result of the call should be an eventual string which is a normalized module name. It is usually a path to the module file or folder from the root of the project. This way it uniquely identifies a module within the project.
Locate: loadRequest → loadRequest
Receives loadRequest object which name property is a normalized module name, and adds an address property to it which holds the resource address. It is called immediately after normalize unless the module is already loaded or loading (the same applies to the rest of the hooks).
Fetch: loadRequest → sourceCodeString
Receives loadRequest object with address property, and returns an eventual string containing the source code of the module.
Translate: loadRequest → ecmaCompliantSourceCodeString
Receives loadRequest object with source property which is a result of the fetch. The purpose of this hook is to translate the source code from another programming language into ECMAScript.
Instantiate: loadRequest → instantiationRequest
Instantiating the translated source. Receives loadRequest with source property as a translated source. Returns an eventual instantiationRequest object which has two required properties. The value of the deps property is an array of strings. Each string is the name of module dependencies. The value of the execute property is a function which the loader will use to create the module.
A module is evaluated during the linking process. First, all of the modules it depends upon are linked and evaluated, and then passed to the execute function. Then the resulting module is linked with the downstream dependencies.
Finale
It is worth mentioning that the current draft of ECMA-262 does not include the specification for module loaders since it was removed in 2014. You can still find it in the archive, it is a very useful resource (if you heard about SystemJS or StealJS you probably know that their implementation was based on that draft). The new draft is now being developed by WHATWG and is not completed yet.
And to recap, we looked at what a module system is, what standards are available for modern web development, then we dove into the loader pipeline and saw how it could be extended. In the next post we will write a simple loader’s plugin for translating CoffeeScript on the fly (correct, no need to precompile, and you can even debug in browser against the original source).
Originally published at Blog • Bitovi.com.
