ClojureScript, WebWorkers & WebGL

ClojureScript users have had the great fortune of gaining access to CSP-type concurrency ever since the announcement of the core.async library a few years ago. And whereas core.async with its channel based philosophy gives us many powerful and deeply transformative abstractions to better structure our code, in many cases it’s often essentially just been used to provide the illusion of concurrent processes in the inherently single threaded environment of the JavaScript VM. This illusion is Good Enough™ and incredibly believable for a large number of lightweight use cases (though overzealous use can have quite an impact on file size!), but if we need to use computationally expensive processes without risking to freeze the browser and destroying the user experience, WebWorkers provide the currently only way out, and have of course their own downsides.

The isolated scope of WebWorkers is one of the more practical downsides, i.e. a WebWorker lives in its own sandbox, needs to be loaded from a separate source file, cannot access the DOM and cannot directly share any data with the main application. The latter can only be achieved via message passing, which in many cases can incur quite an overhead due to needing to clone data (see exceptions below). Of course I realize & appreciate the importance of the various security considerations which caused these constraints, but they do quickly add up to have a dramatic impact on an app’s architecture and the overall development workflow, maybe especially so for ClojureScript.

ClojureScript uses a dual stage compilation strategy, relying on Google’s Closure compiler to produce the final, optimized JS output. For production builds this usually generates a single JS file, and thanks to dead code elimination, this is only containing the actually used parts of the entire application, incl. those of the CLJS runtime itself, as well as any other libraries used (if configured correctly). In addition, the Google Closure compiler has been supporting modular compilation for a long while now, allowing users to split outputs into separate modules and ClojureScript gained access to that feature sometime last year. This means, we can indicate to the compiler which namespaces should end up in which (of the multiple) output files, as well as specify module dependencies. Via its so called “cross module method motion”, the compiler then potentially even further re-arranges functions over the various outputs, e.g. if it can proof that a function is only used by a single module. This is truly splendid and generally works like a charm — unless one wants to use Workers and have them be part of a common code base.

There’re many reasons, both technical and from a UX perspective, why splitting up large code bases for web deployment is an important step to take: We can reduce the initial download size, enable code sharing between modules etc. WebWorkers too can have a positive effect on the overall user experience, generally enabling higher performance due to offering true concurrency (not just an illusion) and avoiding (or at least reducing) stalling UIs and the resulting high blood pressure. But…

Even though ClojureScript & Closure compiler include all the necessary ingredients to enable this modular magic, neither tool can be made aware of the fact that certain namespaces of the common code base are intended to run in a separate scope (i.e. as worker), but still want to make use of other modules and the compiler(s) will therefore produce breaking code when utilizing the full, “advanced” optimization strategy/configuration.

In our ClojureScript workshop last week, we developed a small example discussing some of these pitfalls and I’ve spent some more time afterwards to actually also make it work for production builds.

Live demo. Binary STL mesh loading via WebWorker. The above mesh was generated with the voxel module of thi.ng/geom. The mesh contains ~240k triangles (filesize 11.2MB) and takes 6–8secs to parse. Without workers this would cause an unacceptable UI freeze for the same period…

The example project is a simple STL mesh viewer, using thi.ng/geom’s mesh I/O and WebGL modules, as well as Reagent (just to test with a few more dependencies). Live demo here.

Project configuration

Worker implementation

The worker code is pretty minimal and only responsible for loading, parsing and analyzing the loaded mesh. As mentioned previously, any kind of data can be passed to/from a worker, but usually incurs a deep copy to be created, in order to warrant non-leaking references. Thankfully, there are exceptions and these are especially useful for WebGL-based use cases (or any other use case where binary data is natural & suitable, e.g. asm.js too). In short, data ownership can be literally transferred (instead of copied) to the other party by specifying a list of object references as optional argument to postMessage — here “object” meaning ArrayBuffers. (In case you’re wondering why this is especially suitable for WebGL, it’s because geometry data and other attributes must be defined as typed arrays, hence a perfect match…)

Very important: Since our worker is written in ClojureScript, it needs to import the file base.js (the module containing CLJS etc.). This is done via importScripts. Also note, Workers cannot use the global window object and should use self instead…

Main app

The main app namespace provides various Reagent UI components (incl. a re-usable canvas animation component), the WebGL initialization & update loop, app state handling, initializes the worker and processes messages. The receive-mesh! function is the receiver of mesh data sent from the worker and prepares the mesh for WebGL (the worker itself has no access to it).

Compilation & post-processing

Since the advanced optimizations in the Closure compiler generally completely change the order, naming and presence of things, they will cause havoc in the generated meshworker.js file. Even though we placed the importScripts at the very beginning of the source file, the compiled version (due to x-module motion) has a lot of other code injected/prepended (which is relying on code defined in the base.js file) and therefore is causing errors at runtime. After some experimentation I figured out that this can be avoided by post-processing the JS file and moving the importScripts call to where it belongs: at the beginning of the file. A simple nodejs script can automate this process:

The complete example repo can be found here and has been successfully tested in Chrome (incl. on Android), Firefox, Safari.

http://github.com/thi-ng/ws-ldn-11/day2/ex05b/

Future

WebWorkers are an exciting technology and I think deserve more attention by the larger ClojureScript community. Part of ClojureScript’s rationale was to simplify the development of larger web applications. Support of modular compilation is part of that story and WebWorkers are too.

Last but not least, SharedArrayBuffers in combination with atomic operations (both currently only available in Firefox Nightly) will hopefully soon offer improved flexibility when it comes to better harnessing the resources of contemporary multicore hardware in the browser. It be great to slowly start thinking (in a lazyweb way) if / how ClojureScript could make (better) use of these things.

Show your support

Clapping shows how much you appreciated thi.ng’s story.