Three Routes to Spaghetti-Free Javascript

Published in

Zef.me

7 min readMar 2, 2011

_(If you are familiar with the problems of moving from synchronous to asynchronous programming, feel free to move to the next section.)_

**Update:** A lot of people misunderstood the main issue: [here is another shot at explaining it better.](http://zef.me/3753/planning-ahead-the-async-javascript-problem)

Let’s build a script that determines the titles of a set of URLs. Let’s start simple, we create a function that takes a URL, and returns the title:

var reg = /(.+)<\/title>/mi;</p><p> function getTitle(url) {<br> var body = “<html><title>My title”;
var match = reg.exec(body);
return match[1];
}

In this first prototype we ignore the fetching of the webpage, for now and assume some dummy HTML — for testing purposes. We can now call the function:

getTitle(“http://whatever.com");

and we’ll get back:

“My title”

So far, so good. Now let’s iterate over an array of URLs and get each of their titles:

var urls = [“http://zef.me", “http://google.com",
“http://yahoo.com"];
var titles = [];
for(var i = 0; i < urls.length; i++) {
titles.push(getTitle(urls[i]));
}
console.log(titles);

And the array of resulting titles (all `”My title”`) is printed.

Next, to put in the actual URL fetching part into `getTitle`, we need to make an AJAX call (let’s just ignore the single-source restriction here):

We open an `XMLHttpRequest` and then attach an event listener on the `onreadystatechange` event. When the ready-state changes, we check if it’s now set to `4` (done), and if so, we take the body text, apply our regular expression and return the match.

Or do we?

Note that `return` statement. Where does it return to? Well, it belongs to the event handler function — not the `getTitle` function, so this doesn’t work. The XMLHttpRequest is performed _asynchronously_. The request is set up, an event handler is attached and then the `getTitle` function returns. Then, later at some point, the `onreadystatechange` event is triggered, and the regular expression applied.

So, how do we fix this? Well, we can change our function a little bit:

Now, instead of `return`ing the value, we pass the result to a _callback function_ (the function’s new second argument). When we want to call the function, we have to do it as follows:

getTitle(‘http://bla.com', function(title) {
console.log(“Title: “ + title);
});

That’s annoying, but fair enough.

I suppose we also have to adapt our loop now, too. `getTitle` no longer returns a useful value, so we have to pass a callback function to it. Hmm, how do we do this?

var urls = [“http://zef.me", “http://google.com",
“http://yahoo.com"];
var titles = [];
for(var i = 0; i < urls.length; i++) {
getTitle(urls[i], function(title) {
titles.push(title);
});
}
console.log(titles);

That looks about right. Except, that when running this code the last `console.log` will be executed immediately showing an empty array — because the `getTitle` calls have not finished executing yet. Asynchronous code executes in a different order than the code may suggest.

Shame.

We now have to think — what do we prefer? Do we want to have all the URLs fetched simultaneously, or do they have to be fetched in sequence? Implementing it in sequence is more difficult, so let’s do it in parallel. What we’ll do is add a counter!

As we get back the `getTitle` results we increase the `numFinished` counter, and when that counter has reached the total number of URLs, we’re done — and print the array of titles.

Ugh. Let’s not even look at the code to [fetch these URLs sequentially](http://zef.me/3420/async-foreach-in-javascript).

Centuries of civilization and decades of programming research — and we’re back to this style of spaghetti programming?

There must be ways around this. Indeed, there are — let’s look at three of them.

Route #1: [streamline.js](https://github.com/Sage/streamlinejs)
— — — — — — — — — — — — — — —

[Streamline.js](https://github.com/Sage/streamlinejs) is a simple compiler, implemented in Javascript that enables you to write your code in a synchronous style. The nice thing about Streamline.js is that it operates on regular Javascript and does not add any new keywords — you can keep using your favorite editor and other tools. The only thing streamline.js does, is give the `_` identifier new meaning. Before I demonstrate it, let’s refactor our code slightly. We’ll create a generic `fetchHTML` function:

Now, streamline.js allows us to write our `getTitle` function as follows:

function getTitle(url, _) {
var body = fetchHTML(url, _);
var match = reg.exec(body);
return match[1];
}

You’ll notice the `_` argument there, which represents the callback function. It tells streamline.js that this is an asynchronous function. The next thing you’ll notice is the call to `fetchHTML`, which, although being an asynchronous function, is called as if it’s a regular synchronous function. The difference? The last argument: `_`.

Internally, streamline.js transforms this code to something equivalent to this:

function getTitle(url, _) {
fetchHTML(url, function(body) {
var match = reg.exec(body);
return _(match[1]);
});
}

This transformation is called the [continuation-passing style](http://en.wikipedia.org/wiki/Continuation-passing_style) transformation. We can now keep our loop simple as well:

var urls = [“http://zef.me", “http://google.com",
“http://yahoo.com"];
var titles = [];
for(var i = 0; i < urls.length; i++) {
titles.push(getTitle(urls[i], _));
}
console.log(titles);

Basically the same as our original version, the only difference: an additional `_` argument to `getTitle`.

Not bad huh? [Streamline.js](https://github.com/Sage/streamlinejs) also has some nice functions to enable a parallel version of this code.

**Name:** [streamline.js](https://github.com/Sage/streamlinejs), a Javascript preprocessor (integrates nicely with node.js too)
**License:** MIT

Route #2: [mobl](http://www.mobl-lang.org)
— — — — — — — — — — —

My own project, [mobl](http://www.mobl-lang.org) is a language to rapidly develop mobile web applications. Although it’s not Javascript, the syntax of it scripting language is similar. Since mobl is typed, it is easy for the compiler to infer whether a function is asynchronous or not, which leads to code that is slightly more clean than streamline.js:

function getTitle(url : String) : String {
var body = fetchHTML(url);
var match = reg.exec(body);
return match.get(1);
}

and the loop:

var urls = [“http://zef.me", “http://google.com",
“http://yahoo.com"];
var titles = Array();
foreach(url in urls) {
titles.push(getTitle(url));
}
log(titles);

Like streamline.js, a continuation-passing style is performed by the compiler to produce asynchronous Javascript code.

[Mobl](http://www.mobl-lang.org) is aimed at the mobile web domain, it a whole new language to learn and doesn’t currently support concurrent execution of asynchronous calls. Nevertheless, unlike streamline.js there’s no special `_` variables to pass around.

**Name:** [mobl](http://www.mobl-lang.org), new language, browser only.
**License:** MIT

Route #3: [StratifiedJS](http://onilabs.com/stratifiedjs)
— — — — — — — — — — — — — — —

The most powerful option is [StratifiedJS](http://onilabs.com/stratifiedjs). It extends the Javascript language with various structured concurrency features using a few new language constructs such as `waitfor`, `and`, `or` and `retract`. To fully understand its expressive power, it’s a good idea to have a look at these [excellent interactive OSCON slides](http://onilabs.com/presentations/OSCON2010).

Here’s the code for StratifiedJS:

function getTitle(url) {
var body = fetchHTML(url);
var match = reg.exec(body);
return match[1];
}

and the loop:

var urls = [“http://zef.me", “http://google.com",
“http://yahoo.com"];
var titles = [];
for(var i = 0; i < urls.length; i++) {
titles.push(getTitle(urls[i]));
}
console.log(titles);

As you can see, this code is basically exactly how you’d want to write the code. Compared to our original version, the only thing that changed was adding the `fetchHTML` call — as it should be.

With some effort I was able to capture the Javascript code that is this code fragment is translated to. Here’s the code generated for the `getTitle` function:

var urls, titles, i;
__oni_rt.exseq(this.arguments, this, ‘whatever.js’,
[0, __oni_rt.Nb(function (arguments) {
urls = [“http://zef.me”, “http://google.com",
“http://yahoo.com"];
titles = [];
}, 4), __oni_rt.Seq(0, __oni_rt.Nb(function (arguments) {
i = 0;
}, 8), __oni_rt.Loop(0, __oni_rt.Nb(function (arguments) {
return i < urls.length
}, 5), __oni_rt.Nb(function (arguments) {
return i++
}, 5), __oni_rt.Fcall(1, 6, __oni_rt.Scall(6, function(l){
return [l, ‘push’];
}, __oni_rt.Nb(function (arguments) {
return titles
}, 6)), __oni_rt.Nb(function (arguments) {
return getTitle(urls[i])
}, 6)))), __oni_rt.Nb(function (arguments) {
return console.log(titles)
}, 8)])

What worries me somewhat about this generated code is that it seems rather heavy on the number of functions that’s being generated. Basically every expression is turned into a function passed to another function in the StratifiedJS runtime. This seems rather expensive. I haven’t done any performance benchmarking on this — so maybe it’s not as bad as I think.

Of the three, StratifiedJS is definitely the most flexible and allows you to write the cleanest code. Drawback is that it extends the Javascript language (unlike streamline.js) which could break your current tool chain. In addition, produced code is likely to be slower than the other two solutions.

**Name:** [StratifiedJS](http://onilabs.com/stratifiedjs), extension of Javascript.
**License:** MIT (although source code is only available in a minified version at the moment)

Conclusion
— — — — —

So there you go. Three ways to write clean synchronous code and produce efficient asynchronous Javascript code. The fact is that picking any of these requires a compiler of some kind to be added to your tool chain (although StratifiedJS performs this compilation at run-time), which may or may not be a problem.

A drawback of code generation in any shape or form is debugging. If something goes wrong, the code you’ll be debugging is generated Javascript code. StratifiedJS attempts to include original line numbers when exceptions occur, which helps. A [fork of streamline.js](http://groups.google.com/group/streamlinejs/browse_thread/thread/4da4323f0c2e4a86) attempts to maintain the line numbers in generated code.

In the end it’s all a trade off, a different route would be to use a library like [async.js](https://github.com/caolan/async) that, while not “fixing the language”, gives you an API that enables you to at least write asynchronous code in a more readable manner.

Three Routes to Spaghetti-Free Javascript

Written by Zef Hemel