NPM Packaging
Publishing NPM Packages as Native ES Modules
Optimising NPM Packages
Node.js has had native support for ES modules since version 14, here’s why you should (probably) be using it, how to migrate, and some tips on making things work.
Historically, Node.js used the CommonJS import system, this is the one that looks like const { something } = require('some-module') for imports and module.exports = { something } for exports.
Since 2015, the ECMAScript language that describes the standard interface for javascript has specified what are known as ES modules as the standard for importing and exporting, these are the ones that look like import { something } from 'some-module' for imports and export function something() { /* do something */ } for exports.
Why?
Well, there are some minor reasons, it’s a universal standard and in general it makes sense to follow a universal standard rather than something like CommonJS which is really only used in Node.js. Also, it avoids or reduces the need to transpile your code, meaning the code you run is closer to the code you write, which is nice for debugging. But the most important reason for me is tree shaking.
What is tree shaking?
Tree shaking is a function of webpack which aims to reduce the size of your build assets by removing unused code. Without tree shaking, when you import a function (or constant or whatever) from a module, that whole module will be included in your build assets. If you’re just importing single small functions from a number of large modules this can have a pretty devastating effect on the size of what you deploy.
Why does that matter you ask? Well on the client side this is pretty obvious, large js assets mean more for the client to download before the site starts working. On the server side the same thing applies, larger assets mean larger lambda artefacts or larger Docker images, this means slower cold starts and slower scale up.
What's that got to do with ES modules?
Sensible question.
Basically, for tree shaking to work with webpack, you must be using es imports and exports, and your code must consist of pure functions (ie, no side effects, I’m not going to dig into why you should already be writing pure functions because that’s a whole blog post in its own right).
While you’re probably using es modules for the code you write, the modules you install are probably not published as es modules, which means tree-shaking doesn’t apply. While there isn’t much we can do about all the open-source modules we use, we can certainly make sure the modules we publish internally are published as ES modules.
OK, I’m in, how do I do it?
Fortunately, it’s actually pretty simple. The bare minimum you need to do is:
- Use at least Node.js 14
- Add
"type": "module"to your package.json
Once you’ve done this, Node.js immediately knows your package is using ES modules, and will support your import { whatever } and export const something = ... statements without you needing to run it through babel first. But, there’s a little problem here, Node.js takes an all-or-nothing approach to imports, so now if you’re using require statements anywhere in your code (or config files, or anything else in your package) it’s going to throw an error when it hits that.
.mjs and .cjs files
To help eliminate this all or nothing problem, Node.js uses two new file extensions: .mjs for ES modules, and .cjs for CommonJS modules.
While the module strategy for .js files changes based on the value of the type property in your package.json, .mjs files are always treated as ES modules, and .cjs files always as traditional CommonJS modules. So if you want to keep some scripts or config using CommonJS, just change the file extension. Ideally, to avoid node having to infer the module type from the package.json, you should also rename your ES module js files to .mjs.
Publishing
Before publishing your module again, make sure that the export path in your package.json is still pointing at the right file, for instance if you renamed src/index.js to src/index.mjs you’ll need to update that in your package.json. Remember also to ensure the engines property is set in your package.json to stop people using older unsupported node versions from attempting to install the new version, eg:
"engines": {
"node": ">=14.17.0"
},
"type": "module",
"main": "dist/index.mjs",That’s all it takes.
More Advanced Stuff
If you’re already using ES imports and exports in your code, but not publishing as an ES module, then you’re probably using some tooling to transpile your code. I’m not going to go through all the possibilities here, but I’ll try to mention the setup you need to change for some of the common ones
Babel
Here’s a babel config file from one of our ES modules (quick side note, as you can see, it’s using CommonJS, so its filename in this project is babel.config.cjs instead of babel.config.js):
module.exports = function configureBabel(api) {
api.cache(true); // this tells babel to cache it's transformations, it's pretty good at checking file hashes and invalidating it's cache, but if you have problems with changes not being reflected you can set false here. const presets = [
[
'@babel/preset-env', // this plugin tells babel to transpile your code for a specific runtime environment, we'll use node
{
targets: {
node: '14.17.0', // this means transpile everything that node 14.17 (the version you get in lambda with node14) doesn't support
},
modules: false, // this means imports/exports will not be transformed
},
],
[
'@babel/preset-typescript', // this plugin allows babel to work with typescript (bear in mind it will only transpile it, it doesn't care if you have type errors)
],
]; const plugins = []; return {
presets,
plugins,
};
};
The key line here is in the preset env block: modules: false.
Babel preset env sees this and knows not to transform the imports/exports. Simple as that.
Pro Tip: To make babel write out your files with .mjs extensions instead of .js, just append --out-file-extension '.mjs' to the babel command you use to build your js.
webpack
webpack already supports ES modules properly, and if you’re using babel-loader and you’ve updated your babel config you’re good to go. But, we can improve things. As mentioned earlier, for tree shaking to work, modules must be ES modules, and must be side effect-free. To aid webpack in determining that your module is side effect free, you can add this to your package.json:
"sideEffects": false,
"type": "module",Now, when other people are using your module, webpack is immediately going to know that your module is a safe candidate for tree shaking. A word of caution though, make sure it actually is a side effect free or you’ll end up with weird bugs.
Jest
Eugh. There’s always one thing that just doesn’t work, and with ES modules that one thing (for me) is Jest. If your modules require no transforming, then just disable transformations in jest and it should work. But, if you’re writing in typescript or anything else that needs a transform it's a little bit more of a pain.
Here’s an example jest config from one of our ES modules, I’ll talk you through what it’s doing and why.
import { defaults } from 'jest-config';export default {
moduleFileExtensions: ['cjs', 'mjs', ...defaults.moduleFileExtensions],
// irrelevant stuff removed
transformIgnorePatterns: ['/node_modules/.*\\.js$'],
transform: { '\\.m?[jt]sx?$': 'babel-jest' },
};
- moduleFileExtensions — by default, Jest doesn’t know it should be using cjs and mjs files, so you need to add them to the supported extensions.
- transformIgnorePatterns — by default, Jest doesn’t transform anything inside your
node_modulesdirectory, and since Jest doesn’t work properly with ES modules this means it tends to error when importing es modules from your Node.js modules. To work around this, you can modify the ignore pattern to only ignore js files in yournode_modulesdir, meaning and mjs files will be transformed, letting jest correctly load them. - transform — since you want Jest to use babel to load these files, you’ll need to update the transform option to use babel-jest for mjs as ts, js, jsx, etc.
The Compatibility Problem
Publishing Node.js modules as ES modules is great for the reasons previously discussed, and they work great when you use them within a pure ES modules environment (or when using webpack/babel/etc).
However, unless you’re going to migrate everything to ES modules, you might want to keep your modules compatible with previous versions of Node.js and other environments that don’t support ES modules.
The compatibility problem arises from the require operator not existing in a Node.js process that runs as an ES module, and the import operator not existing in a process that runs as common js. This means that if you try to require an ES module into a CommonJS file (without any webpack magic) you’ll get an error, and if you import a CommonJS file into an ES file you’ll also get an error. Working around this issue to ensure your module can be loaded in either environment is what this post is going to show you how to achieve.
The Compatibility Solution
We’re going to use babel to implement this solution, since it’s pretty much the de facto standard for transforming javascript. For the sake of making examples easy, in this post they’ll be based on a module written in TypeScript (it’s been a long time since I’ve written a module without using TypeScript so I don’t have any examples with js) but the same approach will work with js, jsx, etc.
To make your module run in either a require or an import context, we’re basically going to compile it out twice, once with ES imports, once with CommonJS requires. The file structure will be the same, the content will be the same (except the imports), but the file extensions will be different. For example, given this input:
➜ core git:(main) ✗ tree src
src
├── @types
│ ├── index.ts
│ ├── rails-params-string.ts
│ ├── recursive-rails-params.ts
│ ├── sdk-params.ts
│ └── service-dictionary.ts
├── catalogue-sdk-core.ts
└── index.ts1 directory, 7 files
We’re going to create this output:
➜ core git:(main) ✗ tree dist | grep -v '\.map'
dist
├── @types
│ ├── index.cjs
│ ├── index.d.ts
│ ├── index.mjs
│ ├── rails-params-string.cjs
│ ├── rails-params-string.d.ts
│ ├── rails-params-string.mjs
│ ├── recursive-rails-params.cjs
│ ├── recursive-rails-params.d.ts
│ ├── recursive-rails-params.mjs
│ ├── sdk-params.cjs
│ ├── sdk-params.d.ts
│ ├── sdk-params.mjs
│ ├── service-dictionary.cjs
│ ├── service-dictionary.d.ts
│ ├── service-dictionary.mjs
├── catalogue-sdk-core.cjs
├── catalogue-sdk-core.d.ts
├── catalogue-sdk-core.mjs
├── index.cjs
├── index.d.ts
├── index.mjs1 directory, 35 files
Setting up Your Builds
You can reduce the boilerplate by using the env property in your babel config, but for the sake of readability I’m going to use two separate babel config files and 2 separate compile scripts. One thing I’d strongly recommend though is using the 'import/extensions': ['error', 'always', { ts: 'never' }] eslint rule, this will ensure that for external modules that you import from directly, the file extension will be explicitly set and doesn’t need to be resolved by something else (like webpack). We don’t need to bother with extensions for .ts files since they’ll never be directly loaded by Node.js.
The two babel config files will be babel.config.cjs for the main es module output (and anything else that’s using babel such as jest), and babel.config.compat.cjs for the CommonJS compatibility output.
// babel.config.cjs
module.exports = function configureBabel(api) {
api.cache(true);
const presets = [
[
'@babel/preset-env',
{
targets: {
node: '14.17.0',
},
modules: false, // this means don't transform imports/exports
},
],
['@babel/preset-typescript',],
];
const plugins = [
['babel-plugin-add-import-extension', { extension: 'mjs', replace: true }],
];
return { presets, plugins };
};// babel.config.compat.cjs
module.exports = function configureBabel(api) {
api.cache(true);
const presets = [
[
'@babel/preset-env',
{
targets: {
node: '14.17.0',
},
modules: 'cjs',
},
],
['@babel/preset-typescript'],
];
const plugins = [
['babel-plugin-add-import-extension', { extension: 'cjs', replace: true }],
];
return { presets, plugins };
};
The modules parameter for @babel/preset-env is different, with the ES version having it set to false, telling babel not to transform imports and exports. The CommonJS version has it set to cjs, telling babel to transform imports and exports to CommonJS.
The other difference is the extension parameter passed to babel-plugin-add-import-extension, you’ll need to install this dev dependency. This is telling babel to replace the strings written to imports, exports, and requires the correct file extension for the module system we are targeting.
Next, we’re going to need to create/modify the build scripts in your package.json:
"compile:mjs": "babel src --out-dir dist --extensions '.ts' --out-file-extension '.mjs'",
"compile:cjs": "babel src --out-dir dist --extensions '.ts' --out-file-extension '.cjs' --config-file ./babel.config.compat.cjs",
"compile": "npm run compile:mjs && npm run compile:cjs && tsc --emitDeclarationOnly",So we have two babel scripts, and a main compile script.
The compile:mjs script runs babel, using the default babel.config.cjs file, writing out the files to the dist directory with a .mjsfile extension.
The compile:cjs script is using the babel.config.compat.cjs config and writing out .cjs files.
Running the Build
Let’s run the build and check out what these outputs look like. Given this (truncated) input in src/catalogue-sdk-core.ts:
import { RailParams, RailResponse, RailsData, RailsParams, RailsResponse, Validator } from '@dazn/atlantis-types-esm';
import { SchemaName } from '@dazn/atlantis-types-esm/dist/main/validator.js';
import { AtlantisService, deserialise, serialise, toContentType, toPageType } from '@dazn/atlantis-utils-esm';import { DeserialisedRailsParamString, RecursiveRailsParams, SDKParams, ServiceDictionary, ServiceVersionId } from './@types';export class CatalogueSDKCore {
// ...
}
You’ll get this output in the .mjs file:
import { Validator } from '@dazn/atlantis-types-esm';
import { AtlantisService, deserialise, serialise, toContentType, toPageType } from '@dazn/atlantis-utils-esm';
import { ServiceVersionId } from "./@types/index.mjs";export class CatalogueSDKCore {
// ...
}
And this output in the .cjs file:
Object.defineProperty(exports, "__esModule", {
value: true
});
exports.CatalogueSDKCore = void 0;
var _atlantisTypesEsm = require("@dazn/atlantis-types-esm");
var _atlantisUtilsEsm = require("@dazn/atlantis-utils-esm");
var _index = require("./@types/index.cjs");class CatalogueSDKCore {
// ...
}exports.CatalogueSDKCore = CatalogueSDKCore;
As you can see, the files within the module that are imported have had the file extension replaced (./@types/index was replaced with ./@types/index.mjs or ./@types/index.cjs, note: any files imported from modules will not have their extension replaced, this is correct), the .mjs file has retained the es imports/exports, and the .cjs file has had them replaced by CommonJS requires.
Educating Node.js
This is all good so far, we have the duplicated file structure with the ES version in the .mjs files and the CommonJS version in the .cjs files, but the key to making this compatible with both versions is telling Node.js which files to use based on whether Node.js is running in a module or require mode. You do this by setting some extra parameters in the package.json:
"main": "./dist/index.cjs",
"exports": {
"import": "./dist/index.mjs",
"require": "./dist/index.cjs"
},
"types": "dist/index.d.ts",The main property is there for (really really) old versions of Node.js that don’t understand the newer exports property, since they’re old versions they get the .cjs files. The exports property is what all recent Node.js versions will use. Within the exports property, the import and require props are keywords that Node.js interprets. Essentially they tell Node.js “if the module is being required, use ./dist/index.cjs, if the module is being imported, use the ./dist/index.mjs file”. The result of this is, however, your module is used, the Node.js runtime will be given a version of the code that is compatible with it.
That’s It!
If you’ve gone through this process, your module should now run in pretty much any Node.js version (versions greater than whatever you put in @babel/preset-env anyway). It should work with ES modules where supported, and CommonJS modules where ES modules are not supported. Webpack should tree shake it efficiently, and the bundle sizes should be smaller for all users of your module. If you want to look at a working example of this pattern, you can view the source code for aws-blue-green-toolkit on Github.
