Changing JavaScript one smiley at at time

This narrative is about how a casual email I sent to some people at work ended up causing changes that will affect the entire web browser industry.

I used to write the iOS client for Voxer. It was one of the top social networking/communication app for iOS and Android (these things fluctuate, but for a while it was up in the top 10). It helps people communicate–individually, or in group chats. People like to communicate in lots of different ways so it allows you to share using voice, pictures, or text. The application is available in 50+ countries so it tries to do all of that in as internationally-friendly a way as possible. For text that means using Unicode as the encoding standard.

One of the great things about human communication is that we keep finding so many new ways to do it. One medium is emoji—little picture characters you can embed in your text. The pictures include everything from smiley faces, to buildings, to piles of poo; and they really do help liven up casual communication.

One of the great things about Unicode is that it has lots of room to grow, and the Unicode consortium regularly adds new things to the standard. As part of Unicode 6.0 they added code points for emoji. This was great because previously there were multiple, incompatible, encodings for emoji characters. Making it an open standard was an improvement for the whole tech industry.

One of the great things about Apple is that when they see a standard that makes sense, they adopt it early, and decisively. When iOS 5 was released back in 2011 it included full support for Unicode 6.0 and started using the new standard emoji code points rather than the old incompatible non-standard code points.

It was at this point that the pile of poo hit the fan—neither the emoji for poo, nor the one for a fan, would show up in the Voxer app anymore. While the rest of the technological world had gotten on the full-Unicode bandwagon (sometimes kicking and screaming), the JavaScript world had not. The JavaScript standard still mandated an internal representation of text that could not express all of Unicode. In fact, it could only encode a little more than 65,000 characters in what is known as the Basic Multilingual Plane (BMP). The technology on which Voxer’s servers are based use JavaScript, and as such, was mangling any non-BMP characters because it had no way to represent them. Grrr.

Anyway, early in 2012 I wrote a really detailed email at work explaining the issue to all of our folks who, it turns out, are not big text geeks like I am. The email spelled out the history of Unicode, described how text encodings worked in layman’s terms, and then gave details about the shortcomings of our server technology in this area. Well, by an amazing piece of luck, our CTO ran into Brendan Eich at a conference. Brendan is the guy who invented JavaScript. Lots of talking happened, emails were exchanged, tweets happened, danders were raised, and bugs were filed, etc. It seems that the JavaScript community had been aware of this problem for quite a while, but had never really had much reason to care because, until Unicode 6.0, there were almost no characters outside of the BMP.

As of Unicode 6 there are a lot. There will be a lot more. Pretty much every new character created for Chinese, Korean, Japanese, and lots of other scripts, will go into the extra-BMP space. Plus, you know, you gotta be able to send smiley faces and piles of poo.

I’m happy to say that JavaScript is changing. The technical committee that manages the evolution of ECMAScript, the standard on which JavaScript is based, has said that full Unicode support will be part of ES 6. V8, the JavaScript engine on which node.js is built has already updated, and version 0.8 of node.js includes this shiny new V8 and soon all Voxer users will be able to send each other emoji birthday cakes again.

If you ever think that you’re just a small voice in a big industry, or that big giant international technology consortia don’t care about the needs of users, well, you’re still mostly right. But sometimes, just sometimes, everything works out the way that it should, and smart people work really hard to fix big problems on short schedules, and it makes the world a better place for all of us.