Understanding the hard choice.

I love JavaScript. I love the language. I love the community. I love the modules. In the future JavaScript needs to make a hard choice. I helped create the decision on that choice for Node because I love JavaScript.

I am not the most social creature. I love to talk about technical details and try to see what is coming but this article is more about a journey than those things. A journey that if I took again today, would most likely lead me here again. I want people to understand that it is my love for JavaScript the language that made me choose a file extension even if I get some rather colorful emails.

This post is going to be a little light on some technical topics, because they are subjective. No matter how hard I try to weight decisions; when faces with multiple choices, I must admit everything is subjective at some level.

Let’s rewind for a little bit.

ES modules, they are a standard module system that has beauty and simplicity. Would I say they are flatly better for things than CommonJS? For 90% of cases, yes.

Rewind to a year ago, thats where I started reading; I read specifications, I read blog post, and I read TC39 meeting notes. I decided that Node needed to take a hard look at what implementing ES modules would be like. I wanted to help move both the language and Node forward.

The first thing that stands out in the spec is parsing using a different grammar. This means parsing is very important.

I assumed that we could support them without knowing the file mode prior to parsing them. I was wrong. When I first made the real proposal to support ES modules I thought it would be simply source code detection. I even made a video about it. The double parsing concern looked ok if we optimized for future looking code, checking ES modules first. It would encourage the adoption of modules to make code loading fast even.

Then I started to dig deeper. Quickly, I noted that there were ambiguous source code strings, and we could have false positives. Code running in strict mode that wasn’t supposed to be. I started testing existing modules with implicit strict mode to see how they would stand up if we had false positives.

It was a disaster.

I assumed that the most popular npm modules would be pretty safe. No. Strict mode has some cases where the same code does different things, no errors.

$ node --use_strict
> (function (){return this})() === global
false
>.exit
$ node
> (function (){return this})() === global
true
>.exit

$ node --use_strict
> (function (a) {arguments[0]=”foo”;return a;})(1)
1
> .exit
$ node
> (function (a) {arguments[0]=”foo”;return a;})(1)
‘foo’
> .exit

As you might imagine from these 2 examples, there was a problem with some modules relying on how `this` and `arguments` worked.

Source code parsing was out.

I started looking at a more broad picture of module loading at this point. The specification needed to cover cases that might seem strange, but were present. This moved from a simple parsing problem, to support having an effect on the ecosystem. Browsers, editors, asset pipelines, firewalls, node, shell scripts, etc. would most likely be affected by any decision made after this point.

This was a stressful point, not just in the spec, but in my life. No matter the choice made, there would be a problem because it would affect the ecosystem.

I had to go and do discovery on how people were using `.js`.

JavaScript has several dialects. They are all living in `.js` currently. AMD, CJS, transpiled modules (to some other dialect), browser Scripts, browser Extensions, database queries, embedded systems. Most of the time, using one of these dialects means not being able to live in other places that expect a different dialect. Also, these places generally only support 1 dialect at a time and interop after that point is all userland. UMD is a form of interop but is using runtime feature detection, which is not available to Modules since they have a different parser.

Node will be doing something not present in the mainstream, supporting 2 dialects: CJS and modules.

Well, back to more discovery, what would supporting 2 dialects really mean?

Browsers are using tools like browserify, babel, etc. to take dialects and put them into the browser Script dialect (pretty much just a single flat file). They are taking many files and turning them into a single one of a specific dialect.

This led me to: how would Modules work in the browser using multiple dialects and `<script type=module …></script>`?

On the outside it looks like they don’t, or at least don’t specify it. Loading `src=jQuery.js` would be a bit risky since `jQuery` targets the browser Script dialect and has similar concerns with evaluation in the wrong mode. Thats why there is a Loader Specification. It provides hooks to manipulate fetching, transforms, and execution to browser environments.

That still didn’t fully answer the question though. How could Node support 2 dialects; what was the mechanism to specify the mode.

Look to the JS community if you want to find innovation. Bundlers, transpilers, linters, etc. were using configuration files.

This covered some of the places which would be affected by changes, specifying the file mode explicitly using a configuration file would definitely fall into `package.json` when talking about node core.

However, there is more to it than simply looking at what the JS community has. Questions lingered about things like if the main like entry-point “module” would have downsides. It has some odd edge cases since Node would be supporting both CJS and Modules in the same package. The config would need to list the modes of all the files in a package somehow. Node currently does not require a “package.json” nor a “main” field in that file. How many times had I quickly spun up file to test something in the past month? I knew it was more than a few times.

How would we explain, “if you don’t set these properties” your package will execute in an old mode, even though the source code may still run.

Then there started to be more articles about JS fatigue. Not just from the many different configurations that all sorts of tools were using, but also not knowing exactly what a file was going to do. These files were no-longer self contained.

Several toolchains out there are only made for self contained files. When I open a file, I know what is going to do by the file extension. The only case where this very dubious is `.js`.

I love JavaScript, and `.js` is in the name of so many projects. If files needed to be self contained the only real option was a new file extension. What comes next is what I think summarizes the subjective choices I make.

• I love the language, not the file extension.
• I want JS to thrive not survive.
• I want there to be fewer confusing parts that depend on configuration that I have to track down, and fewer confusing things when I don’t use configuration.
• I want a single module system.
• I want the browser to have easy interop.
• Most of all, I want the transition to be smooth and not make the non-configured way, the deprecated way.

We had come to a point where subjective decisions were all that was left.

What follows is more decision making, more looking at the same proposals of the last 4 months with nuances fixed and cleaned up. We are down to 2, and I am still skeptical of `.js` being used as it leaves the same files being used in multiple ways, but we are trying.

I think the most important thing is to look forward, keep moving, keep struggling to find the right path. Don’t be afraid. Always bet on JS. Tools adapt to configurations and new file extensions come and go. No matter the choice, I know people will be upset, I know my choices will break things for all the proposals. I will keep moving.