I’m happy to announce, as of today, all of my voxel.js modules (as covered in the previous blog post, Six Months of voxel.js) are no longer dependent on transpiling.
Transpilation? What?
As Wikipedia explains, a “transpiler is a type of compiler that takes the source code of a program written in one programming language as its input and produces the equivalent source code in another programming language […] at approximately the same level of abstraction, while a traditional compiler translates from a higher level programming language to a lower level programming language”.
In the context of the web platform, transpiling is often used to compile to JavaScript. But why?
CoffeeScript, The Original Motivating Factor
voxel.js did not originally use CoffeeScript. When @substack and @maxogden initially created voxel.js in 2013, they wrote it in pure JavaScript.
In 2014, I took interest in the voxel.js platform and began to contribute new modules. Being new to JavaScript, wary of its sharp edges (example: == vs ===, strict vs loose typing), I stumbled upon CoffeeScript, “a little language that compiles into JavaScript”. Billing itself as a syntactical improvement over JavaScript, maintaining interoperability but improving on its flaws from being hastily developed in ten days, as the legend goes, it seemed like a good idea at the time to avoid error-prone code.
Especially since JavaScript as a language had seemingly stagnated at the time. Standardized as ECMAScript, version 3 (ES3) in 1999, then a fairly minor ES5 update in 2009, JavaScript as we know it hadn’t significantly changed for more than a decade. Solution: cut the Gordian knot of language evolution blocked on browser adoption by transpiling from any arbitrary language down to the lingua franca of the time, ES5, the universal language accepted by all browsers everywhere.
Enter ES6, Exit CoffeeScript
Then last year in 2015, a major development arose: ES6, aka ECMAScript 2015. The Sixth Edition of ECMAScript, not only is it standardized — unlike Mozilla’s old JavaScript 1.7 extensions, etc. — but it is actually being widely implemented. ES6 is finally here:
V8 5.0, released March 15th, 2016
Google Chrome 50, released in April 20th, 2016
Node.js v6.0.0, released April 26th, 2016
Both Chrome and Node use the V8 engine, supporting more than 90% of ES6 (references: Node, Kangax). The notable missing feature is ES6 modules, but Node and Browserify already support their own CommonJS module system [see below], and even without it ES6 is a major step forward for JavaScript.
Until very recently, ES6 was in an awkward transition phase. You would code in ES6, but transpile down to ES5 using Babel (formerly known as 6to5). Now that modern runtimes support almost all of ES6 natively, this step is no longer needed unless you want to support older versions.
voxel.js requires a modern browser with WebGL and Pointer Lock, originally targeting Chrome 23+ / Firefox 14+. The primary target browser is Chrome (with Firefox a close second), and with Chrome 50 released to the masses, legacy browser support is not much of a concern.
To this end, I began to port of all my modules to use native ES6, removing the transpilation step. Usually this involved removing the coffeeify browser transform, but occasionally babelify. A long process, at last completed:
What advantages does this porting effort bring? This blog post won’t go into too many details, but I’ve compiled a rough list of some of my grievances with transpiling in this list: My Big List of Annoyances. 100% of these issues are eliminated by removing transpilation. The most important is reducing complexity in order to reduce barrier to entry. Fewer moving parts, fewer pieces to break, easier for new developers to contribute.
As for the conversion process itself, I did it all manually for best results. Sometimes you add syntax: parenthesizes around function calls(), semicolons at end of statements;, braces around {bodies}. Sometimes you change punctuation: template strings “#{foo}” becomes `${foo}`. Sometimes you can even remove syntax, method declarations are syntactically cleaner, CoffeeScript’s:
constructor: (game, opts) => {
becomes ES6:
constructor(game, opts) {
just like a normal function declaration, in JavaScript. Some more notes on porting CoffeeScript to ES6.
All in all, I don’t believe I’ve lost much by moving from CoffeeScript or Babel to native ES6, and I’ve gained the advantage of native transpile-free platform support.
Are there zero disadvantages? Well, to be fair benchmarking node-minecraft-protocol shows the native ES6 version is slightly slower than Babel-transpiled ES5. Hopefully as V8 continues to optimize, the performance gap will shrink. For voxel.js at least, I’m not concerned as there is no user-noticeable impact.
That’s about it for this status update, where `npm list` on voxelmetaverse now no longer includes any coffeeify nor babelify. But along the way, there were a couple other interesting projects that came about.
Side Project Detours
cjs4web, Or Not
Removing all traces of transpilers simplifies the build process, but can it be simplified further? Is it possible to develop voxel.js without any build step at all, where all raw source files are included with HTML <script> tags as-is?
To answer this question, it helps to consider only a specific part of the system. voxel.js is built using browserify, which enables using Node.js modules on the web. Distributed through NPM, these modules follow the CommonJS module spec, which essentially is: call require() to import a module, and assign to module.exports to export symbols.
The obvious idea then is to use <script src> to include each module’s source code, with a prelude to implement require() and setup the module global, and a postlude to gather the exported symbols assigned to module.exports:
<script src=“prelude.js”></script>
<script src=“node_modules/ucfirst/ucfirst.js”></script>
<script src=“postlude.js”></script>
The postlude can lookup the included module name by calling document.getElementsByTagName(‘script’), and accessing the .src property of the penultimate <script> currently executed on the page. From there, it can save the exports to a global, keyed by the module path or name. The prelude simply clears the exports, and defines the require() function to lookup saved exports from said global.
Module resolution logic requires additional complexity, but in principle it is feasible. cjs4web demonstrates this example of including CommonJS modules using <script> tags bracketed by prelude and postlude <script>s. But there is a fatal flaw which makes it unworkable: scoping.
CommonJS modules can define their own symbols at the top-level. This includes function foo(), const x = 1, and so on. When you include unmodified CommonJS module source code using the <script> tag, these symbols pollute the browser’s global namespace: window.
So you can try to cleanup the global pollution: save the allowed globals in prelude, and delete all new globals in the postlude. This again, almost works, but per the language spec, you cannot delete function declarations. It is possible to overwrite the function with a variable of the same name then delete it, but then code which called the function can no longer call it!
Browserify’s solution is to wrap modules in a closure. ((require,exports,module) => { …original module code …})(require,exports,module). Then the functions and variables declared within the module do not escape. Of course, this step violates the original goal of not having a build process, the desire to run unmodified NPM modules in the browser, via plain old <script> tags (no XHR/eval trickery here).
As far as I have been able to tell, this problem is unworkable without additional browser support, conceding the need for a build step, or runtime evaluation. Watch for ES6 Modules and the <script type=“module”> tag in the future for a hopefully better answer.
Nodeachrome
Since Chrome 50 and Node v6 both have V8 5.0 (slightly different: Chrome 50.0.2661.94 has V8 5.0.71.39 according to chrome://version/, and node v6.1.0 has 5.0.71.35 according to process.versions.v8), an idea arises. You have two JavaScript runtimes on your system to develop voxel.js, both nearly the same V8 engine. Is it possible to use only one?
Why is Node.js needed to develop voxel.js, anyways? Primarily to run NPM and Browserify, which both use Node.js APIs not available or fully functional in the browser.
An experimental project aiming to change that: https://github.com/deathcap/nodeachrome—porting Node.js APIs to run under a Chrome extension. Works well enough to run browserify and some npm commands (`npm view voxel-engine`). Nodeachrome itself uses browserify but takes it a step further:
http and https Node.js APIs: Network access is allowed to any HTTP host, not only same-origin, due to extra privileges granted to Chrome browser extensions. This is useful to access the NPM registry servers.
fs Node.js API: Native filesystem access using a Google Chrome native host. Rather than a simulated web filesystem, you can use your favorite text editor app to edit the files on your native operating system filesystem.
process and child_process Node.js APIs: Multi-processing using sandboxed iframes. All code runs in iframe sandboxes for eval(), not allowed in Chrome extensions otherwise, but required by browserify’s syntax-error.
Still currently requires Node.js for bootstrapping, but a much smaller area, where most of the code runs in the browser. In principle, with this design it ought to be possible to install Nodeachrome and build voxel.js and other browserify-based projects, but it is not yet there yet for production use.
transpile-c-to-js
Not like asm.js or WebAssembly. Remember the meaning of “transpile”? Retaining high-level code structure, output should be recognizable from input.
Since I couldn’t find a C parser in JavaScript (open to suggestions), investigated using pycparser to transform C syntax to JavaScript syntax. Example of running https://github.com/deathcap/transpile-c-to-js on C source:
#include <stdio.h>
int main(int argc, char **argv) {
printf(“Hello, world!\n”);
return 0;
}
transpiles to JavaScript:
function main(argc, argv)
{
printf(‘Hello, world!\n’);
return 0;
}
with the transpiled JavaScript output you still need to implement your own printf(), possibly wrapping console.log() or document.write() or process.stdout. No implementation is provided by transpile-c-to-js, it is purely a syntactic transformation intended to aid in further manual porting efforts.
The major roadblock in making transpile-c-to-js generally useful is the C preprocessor (as it strips comments and expands macros), but it works well enough to transpile FreeBSD’s bin/ls/ls.c without JavaScript syntax errors. I haven’t found a practical usage yet, but it is kind of neat.
Conclusion
This has been a refinement, cleanup, and modernization of voxel.js modules. Along the way, some fun experimental side project detours.
For better or worse, unlike Minecraft which it was heavily inspired by, the voxel.js project has not yet achieved critical mass as a popular development platform for voxel-style games. Still a small hobbyist project, voxel.js could use all the help it could get. One can hope the removal of transpiling, simplifying the development process to pure native JavaScript, can help towards this goal.
So what are you waiting for? If you’re interested in voxels at all, feel free to pick a project on https://github.com/voxel and fork it today.
Compatible with the voxel-clientmc plugin, embedding a web-based voxel.js client instance to connect to Minecraft, shoutouts to the open source server projects Sponge, Spigot, and Glowstone++.
