How I Built Emojitracker

Adventures in Unicode, Real-time Streaming, and Media Culture

Emojitracker was one of those projects that was supposed to be a quick weekend hack but turned into an all-consuming project that ate up my nights for months. Since its launch in early July, Emojitracker has processed over 1.8 billion tweets, and has been mentioned in approximately a gajillion online publications.

Emojitracker wasn’t my first megaproject, but it is definitely the most complex architecturally.

While the source code for emojitracker has been open-source since day one, the technical concepts are complex and varied, and the parts of the code that are interesting are not necessarily obvious from browsing the code. Thus, rather than a tutorial, in this post I intend to write about the process of building emojitracker: the problems I encountered, and how I got around them.

This is a bit of a brain dump, but my hope is it will be useful to others attempting to do work in these topic areas. I have benefited greatly from the collective wisdom of others in the open-source community, and thus always want to try to do my best to contribute back domain knowledge into the commons.

This post is long, and is primarily intended for a technical audience. It details the origin story and ideas for emojitracker, the backend architecture in detail, frontend client issues with displaying emoji and high-frequency display updates, and the techniques and tools used to monitor and scale a multiplexed real-time data streaming service across dozens of servers with tens of millions of streams per day (on a hobby project when you don’t have any advance warning!).


Prologue: Why Emoji?

These fingers wrote a lot of emoji code, they earned it.

I’ve also always had a soft spot for emoji. My friends and colleagues know that emoji makes an appearance in many aspects of my life, including my wifi network, LinkedIn recommendations, and domain names. I even once signed a part-time employment contract stipulating emoji was my native language and all official notices be provided to me that way (which I don’t necessarily endorse). Oh, and then there’s the emoji nail art (which I do endorse).

I’d been playing around with the idea of realtime streaming from the Twitter API on a number of previous projects (such as goodvsevil, which was the spiritual predecessor to emojitracker), and I was curious about seeing how far I could push it in terms of number of terms monitored. At 842 terms to track, emoji seemed like a prime candidate.

Emoji are also a great way to get insight to the cultural zeitgeist of Twitter: the creative ways in which people appropriate and use emoji symbols is fascinating, and I hoped to be able to build a lens that would enable one to peer into that world with more detail.

And finally, (and quite foolishly) emoji seemed simple at the time. Normally I try to pick hacks that I can implement pretty quickly and get out into the world within a day or two. Boy, was I wrong in this case. Little did I know how complex emoji can be… This post is a testament to the software development journey emoji brought me on, and the things I learned along the way.

Background Understanding: Emoji and Unicode

The history of Emoji has been written about in many places, so I’m going to keep it brief here and concentrate more on the technical aspects.

TLDR: Emoji emerged on feature phones in Japan, there were a number of carrier specific implementations (Softbank/KDDI/Docomo), each with its own incompatible encoding scheme. Apple’s inclusion of Emoji on the iPhone (originally region-locked to Asia but easily unlocked with third-party apps) led to an explosion in global popularity, and now Emoji represents the cultural force of a million voices suddenly crying out in brightly-colored pixelated terror.

For some background music, watch Katy Perry demonstrate why she should have been a primary delegate on the Unicode Consortium Subcommittee on Emoji at http://youtu.be/e9SeJIgWRPk.

But for the modern software developer, there are a few main things you’ll need to know to work with Emoji. Things got a lot better in late 2010 with the release of Unicode 6.0… mostly. The emoji glyphs were mostly standardized to a set of Unicode codepoints.

Now, you may be thinking: “Wait, standards are good, right? And why do you say ‘mostly’ standardized, that sounds suspicious…”

Of course, you’d be correct in your suspicions. Standardization is almost never that simple. For example, take flags. When time came to standardize Emoji codepoints, everyone wanted their country’s flag added to the original 10 in the Softbank/DoCoMo emoji. This had the potential to get messy fast, so instead what we ended up with were 26 diplomatically-safe “Regional indicator symbols” set aside in the Unicode standard. This avoided polluting the standard with potentially hundreds of codepoints that could become quickly outdated with the evolving geopolitical climate, while preserving Canada’s need to assert their flag’s importance to the Emoji standardization process:

These characters can be used in pairs to represent regional codes. In some emoji implementations, certain pairs may be recognized and displayed by alternate means; for instance, an implementation might recognize F + R and display this combination with a symbol representing the flag of France.

Note the standards-body favorite phrases “CAN BE” and “MAY BE” here. This isn’t a “MUST BE,” so in practice, none of the major device manufacturers have actually added new emoji art flags, infuriating iPhone-owning Canadians every July 1st:

https://twitter.com/withloveclaudia/statuses/351744535291887616

For a detailed and amusing exploration of this and other complex issues surrounding the rough edges of Unicode, I highly recommend Matt Mayer’s “Love Hotels and Unicode” talk, which was invaluable in helping my understanding when parsing through these issues.

For these double-byte emoji glyphs, the popular convention is to be represent them in ID string notation with a dash in between the codepoint identifiers, such as 1F1EB-1F1F7.

This of course makes the life of someone writing Emoji-handling code more difficult, as pretty much all the boilerplate you’ll find out there assumes a single Unicode code point per character glyph (since after all, this was the problem that Unicode was supposed to solve to begin with).

For example, say you want to parse and decode an emoji character from a UTF-8 string to identify its unified codepoint identifier. Conventional wisdom would be that this a simple operation, and you’ll find lots of sample code that looks like this:

# return unified codepoint for a character, in hexadecimal
def char_to_unified
(c)
c.unpack("U*").first.to_s(16)
end

If you have a sharp eye, you’ll probably notice the danger-zone of using first() to convert an array into a string: assuming we’re always going to get one value back from the unpack() since we only sent one character in. And in most cases, it will of course work fine. But for our strange double-byte emoji friends, this won’t work, since that unpack() operation is actually going to return two values, the second of which we’ll be ignoring. Thus, if we pass in the American Flag emoji character, we’ll get back 1f1fa—which represents the rather boring on its own REGIONAL INDICATOR SYMBOL U:

Figure 1: Not the American Flag.

So instead, we have to do some string manipulation hijinks like this:

# return unified codepoint for a character, in hexadecimal.
#  — account for multibyte characters, represent with dash.
#  — pad values to uniform length.

def char_to_unified
(c)
c.codepoints.to_a.map {|i| i.to_s(16).rjust(4,'0')}.join('-')
end

Now, char_to_unified() on a UTF-8 string containing the American Flag emoji will return the properly patriotic value 1f1fa-1f1f8.

Figure 2: The land of the free, and the home of the brave.

Victory!


Surprisingly, there wasn’t a good Ruby library in existence to handle all this (most existing libraries concentrating on encoding/decoding emoji strictly in the :shorthand: format).

Thus, I carved that portion of the work in emojitracker out into a general purpose library now released as its own open source project: emoji_data.rb. It handles searching the emoji space by multiple values, enumeration, convenience methods, etc. in a very Ruby-like way.

For example, you can do the following to find the short-name of all those pesky double-byte Emoji glyphs we mentioned:

>> EmojiData.all.select(&:doublebyte?).map(&:short_name)
=> [“hash”, “zero”, “one”, “two”, “three”, “four”, “five”, “six”, “seven”, “eight”, “nine”, “cn”, “de”, “es”, “fr”, “gb”, “it”, “jp”, “kr”, “ru”, “us”]

For more examples, check out its README. This library is consistently used across almost all of the different software projects that make up Emojitracker, and hopefully will be useful for anyone else doing general purpose Emoji/Unicode operations!

Emojitracker Backend Architecture

Here’s the overall architecture for Emojitracker in a nutshell: A feeder server receives data from the Twitter Streaming API, which it then processes and collates. It sends that data into Redis, but also publishes a realtime stream of activity into Redis via pubsub streams. A number of web streamer servers then subscribe to those Redis pubsub streams, handle client connections, and multiplex subsets of that data out to clients via SSE streaming.

We’ll talk about all of these components in detail in this section.

Feeding the Machine: Riding the Twitter Streaming API

If you’re doing anything even remotely high volume with Twitter, you need to be using the Streaming APIs instead of polling. The Streaming APIs allow you to create a set of criteria to monitor, and then Twitter handles the work of pushing updates to you whenever they occur over a single long-life socket connection.

In the case of Emojitracker, we use our EmojiData library to easily construct an array of the Unicode chars for every single Emoji character, which we then send to the Streaming API as track variables for status/filter. The results are easy to consume with a Ruby script utilizing the TweetStream gem, which abstracts away a lot of the pain of dealing with the Twitter Streaming API (reconnects, etc) in EventMachine.

From this point it’s simple to have an EventMachine callback that gets triggered by TweetStream every time Twitter sends us a matching tweet. It’s important to note that the Streaming API doesn’t tell you which track term was matched, so you have to do that work yourself by matching on the content of the tweet.

Also, keep in mind that it’s entirely possible (and in our case, quite common!) for a tweet to match multiple track terms—when this happens, the Twitter Streaming API is still only going to send it to you once, so it’s up to you to handle that in the appropriate fashion for your app.

Then, we simply increment the count for each emoji glyph contained in the tweet (but only once per glyph) and also push out the tweet itself to named Redis pubsub streams (more details on the structure for this in the next section).

The JSON blob that the Twitter API sends for each tweet is pretty massive, and at a high rate this will get bandwidth intensive. The feeder process for Emojitracker is typically receiving a full 1MB/second of JSON data from Twitter’s servers.

Since in our cases we’re going to be re-broadcasting this out at an extremely high rate to all the streaming servers, we want to trim this down to conserve bandwidth. Thus we create a new JSON blob from a hash containing just the bare minimum to construct a tweet: tweet ID, text, and author info (permalink URLs are predictable and can be recreated with this info). This reduces the size by 10-20x.

As long as you drop-in a performant JSON parsing engine (I use and highly recommend Oj), you can do all this parsing and recombining with relatively low server impact. Swap in hiredis for an optimized Redis driver and things can be really fast and efficient: the feeder component for Emojitracker is acting upon ~400-500 tweets-per-second at peak, but still only operates at ~10-12% CPU utilization on the server it runs on, in MRI Ruby 1.9.3. In reality, network bandwidth will be the biggest constraint once your code is optimized.


Data Storage: Redis sorted sets, FIFO, and Pubsub streams

Redis is an obvious data-storage layer for rapidly-changing and streaming data. It’s super fast, has a number of data structures that are ideally suited for this sort of application, and additionally its built-in support for pubsub streaming enables some really impressive ways of shuffling data around.

For emojitracker, the primary piece of data storage we have is a set of emoji codepoint IDs and their respective counts. This maps very well to the Redis built-in data structure Sorted Set, which conveniently maps strings to scores, and has the added benefit of making it extremely fast to query that list sorted by the score. From the Redis documentation:

Redis Sorted Sets are, similarly to Redis Sets, non repeating collections of Strings. The difference is that every member of a Sorted Set is associated with score, that is used in order to take the sorted set ordered, from the smallest to the greatest score. While members are unique, scores may be repeated. With sorted sets you can add, remove, or update elements in a very fast way (in a time proportional to the logarithm of the number of elements). Since elements are taken in order and not ordered afterwards, you can also get ranges by score or by rank (position) in a very fast way.

This makes keeping track of scores and rank trivially easy. We can simply fire off ZINCRBY increment commands to the set for the equivalent emoji codepoint ID every time we see a match—and then call ZRANK on an ID to find out it’s current position, or use ZRANGE WITHSCORES to get the entire list back in the right order with the equivalent numbers for display.

This gives us an easy way to track the current score and ranking, but we want to stream updates in realtime to clients, so what we really need in addition is way to send those update notifications out. Thankfully, Redis PUBLISH and SUBSCRIBE is essentially perfect for that.

With Redis Pubsub streams, the feeder can simply publish any updates to a named stream, which an client can subscribe to to receive all messages. In Emojitracker, we publish two types of streams:

  1. General score updates. Anytime we increment a score for an Emoji symbol, we also send an activity notification of that update out to stream.score_updates.
  2. Tweet streams. 842 different active streams for these (one for each emoji symbol). This sounds more complex than it is—in Redis, streams are lightweight and you don’t have to do any work to set them up, just publish to a unique name. For any matching Tweet, we just publish our “small-ified” JSON blob to the equivalent ID stream. For example, a tweet matching both the dolphin and pistol emoji symbols would get published to the stream.score_updates.1f42c and stream.score_updates.1f52b streams.
Illustration: crossing 842 pubsub streams with a single PSUBSCRIBE statement in Redis.

Clients can then subscribe to whichever streams they are interested in, or use wildcard matching (PSUBSCRIBE stream.score_updates.*) to get the aggregate of all tweet updates.

While this live stream of tweets in Emojitracker is mostly powered by the aforementioned Pubsub streams, there are cases where they won’t work. For example, when a new web client connects to a detail stream it’s necessary to “backfill” the most recent 10 items for display so that the client starts with some data to show the user (especially on the less frequently used emoji symbols).

A totally ridiculous illustration of a FIFO queue I found on the web. I decided it required some emojification.

Redis doesn’t have a built-in concept of a fixed-size FIFO queue (possibly more accurately described as a fixed-size evicting queue?), but this is easy to emulate by using LPUSH and LTRIM. Push to one side of a list, and then immediately trim from the other to maintain the fixed length. Like most things in Redis, it doesn’t matter if these commands come out of order, it will balance out and the overall size of the list will remain relatively constant. Easy-peasy.

Putting it all together, here’s the relevant section of source code from the Ruby program that feeds Redis from the Twitter streaming API (I included the usage of the aforementioned EmojiData library to do the character conversion):

 matches = EmojiData.chars.select { |c| status.text.include? c }
matches.each do |matched_emoji_char|
# get the unified codepoint ID for the matched emoji char
cp = EmojiData.char_to_unified(matched_emoji_char)
REDIS.pipelined do
# increment the score in a sorted set
REDIS.ZINCRBY 'emojitrack_score', 1, cp
# stream the fact that the score was updated
REDIS.PUBLISH 'stream.score_updates', cp
# for each emoji char, store most recent 10 tweets in a list
REDIS.LPUSH "emojitrack_tweets_#{cp}", status_json
REDIS.LTRIM "emojitrack_tweets_#{cp}",0,9
# also stream all tweet updates to named streams by char
REDIS.PUBLISH "stream.tweet_updates.#{cp}", status_json
end
end

It’s common knowledge worth repeating that Redis is highly performant. The current instance powering Emojitracker routinely peaks at 2000-4000 operations/second, and only is using ~3.98MB of RAM.


Pushing to Web Clients: Utilizing SSE Streams

When thinking about streaming data on the web, most people’s thoughts will immediately turn to WebSockets. It turns out, if you don’t need bidirectional communication, there is a much simpler and well suited technology that accomplishes this over normal HTTP connections: Server-Sent Events (SSE).

I won’t go into detail about the SSE protocol (the above link is a great resource for learning more about it), instead I’ll just say it’s trivially easy to handle SSE in Javascript, for example the full logic for subscribing to an event source and passing events to a callback handler can be accomplished in a barely more than a single line of code. The protocol will automatically handle reconnections, etc. The more interesting aspect for us is how we handle this on the server side.

Each web streamer server maintains two connection pools:

  1. The raw score stream — anything connected here is going to get everything rebroadcast from the score update stream, and everyone gets the same thing. Pretty simple.
  2. The tweet detail updates queue is more complex. We use a connection wrapper that maintains some state information for each client connected to the stream. All web clients receiving tweet detail updates from the streaming server are actually in the same connection pool, but when they connect they pass along as a parameter the ID of the emoji character they want updates on, which gets added to their wrapper object as tagged metadata. We later use this to determine which updates they will receive.

There are typical Sinatra routes that handle incoming stream connections, and essentially all they do is use stream(:keep_open) to hold the connection open, and then add the connecting client’s information to to the connection pool. When the client disconnects, Sinatra removes it from that pool.

In order to populate the SSE streams on the server side, we need to get the data out of Redis to pass along. Each web streamer server spawns two independent event-propagation threads, each of which issues a SUBSCRIBE to a Redis stream. Not surprisingly, these are the two types of streams we mentioned in the previous section: 1.) The overall score updates stream, and 2.) a wildcard PSUBSCRIBE representing the aggregate of all individual tweet streams.

Each thread then processes incoming events from the Redis streams, iterating over every client in the connection pool and writing data out to it. For the raw score updates, this is just a simple iteration, for the tweet details, each wrapped connection in the pool has it’s tag compared to the event ID of the current event, and is only written to in the case of a match.

The end result is a relatively efficient way to stream updates out to many clients simultaneously, even though they may be requesting/receiving different data.


Performance Optimizations for High Frequency SSE Streams

SSE is great, but when you start to approach hundreds of events per second, raw bandwidth is going to become a concern. For Emojitracker, we needed to turn to a number of performance enhancements were necessary to reduce the bandwidth of the stream updates so that people without super-fat pipes could play along.

Note: both of these optimizations are probably overkill unless you are handling at least tens if not hundreds of events per second, but in extremely high-frequency applications they are the only way to make things possible.

Trim the actual SSE stream format as much as possible.
Every character counts here. SSE streams can’t be gzipped, so you need to be economical with your formatting. For example, the whitespace after the colon in DATA: is optional. One character multiplied by potentially hundreds of times per second ends up being quite a bit over time.

Consider creating a cached “rollup” version of the stream that aggregates events.

You’re never going to need to update your client frontend more than 60 times per second, as that’s above what humans can perceive. That seems pretty fast, but in Emojitracker’s case, we actually are high frequency enough that we typically have many score updates occur in every 1/60th of a second ticket.

Thus, instead of rebroadcasting each of these events out immediately upon receiving them from the Redis pubsub stream, each web stream holds them in an in-memory queue which we expunge in bulk 60 times per second, rolling up the number of events that occurred for each ID in that timeframe.

Therefore, where normally in one 1/60th of a second tick we would send this:

data: 2665  \n\n
data: 1F44C \n\n
data: 1F44F \n\n
data: 1F602 \n\n
data: 2665 \n\n
data: 1F60B \n\n
data: 1F602 \n\n

We can instead send this:

data:{"2665":2,"1F44C":1,"1F44F":1,"1F602":2,"1F60B":1}\n\n

The size savings from eliminating the redundant data headers and repeat event IDs is nontrivial at scale (remember, no gzipping here!). You can compare and see the difference in action yourself by curl-ing a connection to emojitracker.com at /subscribe/raw and /subscribe/eps.

Even though in emojitracker’s case we go with 60eps for maximum disco pyrotechnics, in many cases you can likely get away with far more aggressive rollups, and broadcast at 30eps or even 5-10fps while still maintaining the user experience of full-realtime updates.


Gotcha: Many “cloud” environments don’t properly support this (and a workaround)

The crux: after building all this in development environment, I realized it wasn’t quite working correctly in production when doing load testing. The stream queue was filling up, getting bigger and bigger, never reducing in size. After much spelunking, it turned out that the routing layer used by many cloud server providers prevents the web server from properly seeing a stream disconnection on their end. In an environment where we are manually handling a connection pool, this is obviously no good.

My solution was to hack in a REST endpoint where clients could send an asynchronous “I just disconnected” post— the stream server would then manually expunge the client record from the pool.

I wasn’t 100% satisfied with this solution— I figured some portions of clients would disconnect without successfully transmitting the cleanup message (flakey net connections for example). Thus, the stream server also sweeps for and manually disconnects all stream connections after they hit a certain stream age. Clients that were actually active will then automatically reestablish their connection. Again, it’s ugly, but it works. I maintained the appearance of a continuous stream without stutter by reducing the EventSource reconnect delay significantly.

These were, of course, temporary hacks that were far less efficient in terms of extra HTTP requests (albeit ones that managed to carry emojitracker through it’s peak traffic). Thankfully, they are no longer needed. Very recently, Heroku finally rolled out labs support for Websockets which also fixes the underlying routing issues affecting SSE, thus removing the need for the workaround. (Thankfully, I made my workaround hacks enabled via a config variable, so once I added websockets support to my dynos I was able to quickly disable all those hacks and see everything worked fine.)

(You may be thinking that with all these workaround hacks it wasn’t worth hosting at Heroku at the time, and I should have just used my own conventional dedicated server. However, you’ll see later why this would have been a bad idea.)


With all these changes, one might wonder how I monitored the streaming connection pool to see how things were working. The answer: a custom admin interface.


Not crossing the streams: The admin interface

When attempting to debug things, I quickly realized that tailing a traditional log format is a really terrible way to attempt to understand what’s going on with long-lived streams. I hacked up a quick web interface showing me the essential information for the connection pools on a given web server: how many open connections and to whom, what information they were streaming, and how long those connections had been open:

Part of the stream admin interface for one of the web dynos, on launch day.

The stream admin interface is actually open to the public, so you can mess around with it yourself.

Having an admin interface like this was absolutely essential to being able to visualize debug the status of streaming pools. From just watching the logs, there’s no way I would have noticed the connection pool problem in the previous section.


Frontend Architecture

For the most part, there is nothing that surprising here. Consuming a SSE stream is a fairly simple endeavor in Javascript, with widespread browser support. However, there were a number of “gotchas” with secondary functionality that ended up being somewhat complex.

Rendering Emoji Glyphs

Spoiler alert: sadly, most web browsers don’t support emoji display natively (Google, get on this! Forget Google+, we want emoji in Chrome!). Thankfully, you can utilize Cal Henderson’s js-emoji project to sniff the browser and either serve native emoji unicode or substitute in images via JS for the other browsers.

For that though, you still need to host a few thousand images for all the different emoji symbols. If you’re going to want to display in more than one resolution, multiply that by 5x. Too add to the problems, most of the existing emoji graphic sets out there (such as a the popular gemoji), have unoptimized PNGs and are missing many common display resolutions.

I wanted to solve this problem once and for all, so I created emojistatic.

Emojistatic is a hosted version of the popular gemoji graphic set, but adds lots of optimizations. It has multiple common sizes, all losslessly compressed and optimized, hosted on GitHub’s fast infrastructure for easy access.

It does more too, out of necessity. There are unfortunately many other problems inherent in display emoji beyond just swapping in appropriate images. I’ll discuss some of them here, and try to show what the emojistatic library does to help address them.


Image combination to reduce HTTP requests
Swapping in images is great in some instances, but what if you are displaying a lot of emoji? For example, in emojitracker’s case, we are displaying all 862 emoji glyphs on the first page load, and making 862 separate HTTP requests to get the images would be crazy.

Image via emojinal art gallery.

Therefore, I built-in automatic CSS spritesheet generation to emojistatic. I used the embedded data-URI CSS sheet technique instead of a spritesheet, because shuffling around literally thousands of copies of a 1MB image in memory could have grave performance implications. In order to facilitate this, I ended up spinning off another open-source tool, cssquirt, a Ruby gem to embed images (or directories of images) directly into CSS via the Data URI scheme.

In order to get this to work with js-emoji, I had to fork it to add support to it for using the data-URI technique instead of loading individual images. The changes are in a pull-request, but until the maintainer accepts it (nudge nudge), you’ll unfortunately have to use my fork.


Native emoji display: the cake is a lie
What a pain. At least it must be easier on those web clients that support Emoji fonts natively, right? Right?!?! If we just stick to Safari on a fancy new OSX 10.9 install, surely Apple’s love for technicolor cuteness will save us? …Unfortunately, no. (Insert loud sigh) Can’t anything ever be simple?

What doesn’t work properly? Well, if you have a string with mixed content (for example, most tweets containing both words and emoji characters), and you specify a display font in CSS, characters that have non-Emoji equivalents in their font-face will default to their ugly, normal boring versions. So you get a ☁︎ symbol instead of the lovely, fluffy emoji cloud the person used in their original tweet.

If you try to get around this on a Mac by forcing the font to AppleColorEmoji in CSS, you will have similarly ugly results, as the font actually contains normal alphanumeric characters, albeit with weird monospace formatting.

Native-rendering of an English tweet containing Emoji in Safari 7.0 on MacOSX 10.9.
To get around this problem, I stumbled along the technique of creating a Unicode-range restricted font-family in CSS, which will let us instruct the browser to only use the AppleColorEmoji font for those particular 842 emoji characters.

Listing out all 842 codepoints would work, but would result in a bulky and inefficient CSS file. Unfortunately, a simple unicode-range won’t work either, as Emoji symbols are strewn haphazardly across multiple locations in the Unicode spec. Thus, to generate the appropriate ranges in an efficient manner for emojistatic, we turn again to our EmojiData library, using it to find all sequential blocks of Emoji characters greater than 3 in size and compressing them to a range. Go here to examine the relevant code (it’s a bit large to paste into Medium), or just check out the results:

>> @emoji_unicode_range = Emojistatic.generate_css_map
=> "U+00A9,U+00AE,U+203C,U+2049,U+2122,U+2139,U+2194-2199,U+21A9-21AA,U+231A-231B,U+23E9-23EC,U+23F0,U+23F3,U+24C2,U+25AA-25AB,U+25B6,U+25C0,U+25FB-25FE,U+2600-2601,U+260E,U+2611,U+2614-2615,U+261D,U+263A,U+2648-2653,U+2660,U+2663,U+2665-2666,U+2668,U+267B,U+267F,U+2693,U+26A0-26A1,U+26AA-26AB,U+26BD-26BE,U+26C4-26C5,U+26CE,U+26D4,U+26EA,U+26F2-26F3,U+26F5,U+26FA,U+26FD,U+2702,U+2705,U+2708-270C,U+270F,U+2712,U+2714,U+2716,U+2728,U+2733-2734,U+2744,U+2747,U+274C,U+274E,U+2753-2755,U+2757,U+2764,U+2795-2797,U+27A1,U+27B0,U+27BF,U+2934-2935,U+2B05-2B07,U+2B1B-2B1C,U+2B50,U+2B55,U+3030,U+303D,U+3297,U+3299,U+1F004,U+1F0CF,U+1F170-1F171,U+1F17E-1F17F,U+1F18E,U+1F191-1F19A,U+1F201-1F202,U+1F21A,U+1F22F,U+1F232-1F23A,U+1F250-1F251,U+1F300-1F31F,U+1F330-1F335,U+1F337-1F37C,U+1F380-1F393,U+1F3A0-1F3C4,U+1F3C6-1F3CA,U+1F3E0-1F3F0,U+1F400-1F43E,U+1F440,U+1F442-1F4F7,U+1F4F9-1F4FC,U+1F500-1F507,U+1F509-1F53D,U+1F550-1F567,U+1F5FB-1F640,U+1F645-1F64F,U+1F680-1F68A,U+1F68C-1F6C5"

This is then dropped into an appropriately simple ERB template for the CSS file:

@font-face {
font-family: 'AppleColorEmojiRestricted';
src: local('AppleColorEmoji');
unicode-range: <%= @emoji_unicode_range %>;
}
.emojifont-restricted {
font-family: AppleColorEmojiRestricted, Helvetica;
}

When we then use the resulting .emojifont-restricted class on our webpage, we can see the improved results:

Same example, but custom font range saves the day. (try demo in your own browser: http://codepen.io/mroth/pen/cpLyK)

Yay! But unfortunately, this technique isn’t perfect. Remember those double-byte Unicode characters we talked about earlier? You may have noticed we rejected them in the beginning of our unicode-range generation algorithm. Well, turns out that they are obscure enough that there is no way to represent them in standard CSS unicode-range format. So by doing this, we do lose support for those few characters represented in a mixed string, and we can actually only display 821 of the emoji glyphs.Win some, lose some, eh? I’ve looked long and hard without being able to find a solution, but if anyone has a secret workaround for this, please let me know! For now though, this seems to be the best case scenario.

Keeping it all up to date: chained Rake file tasks
Keeping all these assets up to date in emojistatic could be a pain in the rear when something changes. For example,add one emoji glyph image, you’ll not just need new optimized versions of it, but also to generate new versions of the rollup spritesheets, minify and gzipped versions of those, etcetera. Rake file are incredibly powerful, because they allow you to specify the dependency chain, and then are smart enough to rebuild just the necessary tasks for any change. A full run of emojistatic can take 30-40 minutes from a fresh state (there’s a ton of image processing that happens), but subsequent changes occur in seconds. Once you get it working, it feels like magic.

Going into the detail of complex Rake file tasks is beyond the scope of what I want to cover in this blog post, but if you do anything at all like this, I highly recommend watching Jim Weirich’s Power Rake talk, which was immensely helpful for me in grokking proper usage for this technique.


Frontend Performance

Image from http://ftw.usatoday.com/2013/09/emoji-sports-art-is-the-best-kind-of-art

It took lots of attempts to figure out how to get so many transitions to occur on the screen without slowdown. My goal was to have emojitracker work on my iPad, but early versions of the site were bringing my 16GB RAM quad-core Core i7 iMac to its knees begging for mercy.

Crazy, since it’s just a webpage, right? The DOM really wasn’t meant for handling this many operations at this speed. Every single manipulation had to be optimized, and using jQuery for DOM manipulation was out of the question — everything in the core update event loop needed to be written in pure Javascript to shave precious milliseconds.

Beyond that though, I looked at a number of different techniques to try to optimize the DOM updates and visual transitions (my good pal and Javascript dark wizard Jeff Tierney was extremely helpful with this). Some of the comparisons and optimizations we examined were:

  • Utilizing explicit CSS animations vs. specifying CSS transitions. (using Javascript based animation was entirely out of the question as we needed native rendering to get GPU acceleration.)
  • Different methods of force triggering the transition animation to display:replacing an entire element vs. forcing a reflow vs. using a zero length timeout.
  • Maintaining an in-memory cache of DOM elements as a hash, avoiding repeated selection.

And of course, all of the various combinations and permutations of these things together (as well as the 60eps capped event stream mentioned in the backend section versus the full raw stream). Some might work better with others, and vice versa. The end user’s computer setup and internet connection also would play a factor in overall performance. So what combination would get us the absolute best average frames-per-second display in most environments?

To test this, all methods are controlled via some variables at the beginning of our main Javascript file, and the logic for each remains behind branching logic statements in the code. As a result, we can switch between any combination of methods at runtime.

A special benchmark page can be loaded that has test metrics visible with a benchmark button. The additional JS logic on that page basically handles stopping and restarting the stream for a distinct period of time using every possible combination of methods, while using FPSMeter.js to log the average client performance.

Menu bar during a benchmark performance test, showing the current animation method and FPS.

Upon completion, it creates a JSON blob of all the results for viewing, with a button that will send the results back to our server for collation.

Benchmark results as JSON blob.

This gave me an easy way to ask various people to easily yet exhaustively test how it performed on their machines in a real world way, while getting the results back in a statistically relevant fashion.

If you’re interested, you can check out the the full test suite logic in the source.

(Oh, and by the way, the overall winner in this case ended up being using the capped stream, cached elements and zero-length timeouts. This is probably not what I would have ended up choosing based on testing on my own machine and gut intuition. Lessons learned: test your assumptions, and sometimes the ugly hacks work best.)

In the future, I’m almost certain I could achieve better performance by using Canvas and WebGL and just drawing everything from scratch (ignoring the DOM entirely), but that will remain an exercise for another day—or for an intrepid open source contributor who wants to send a pull request!


Deploying and Scaling

The first “soft launch” for Emojitracker was on the Fourth of July, 2013. I had been working on emojitracker for months, getting it to work had consumed far more effort than I had ever anticipated, and I just wanted to be done with it. So I bailed on a party in Red Hook, cabbed it back up to North Brooklyn, and removed the authentication layer keeping it hidden from the public pretty much exactly as the fireworks displays began.

https://twitter.com/mroth/status/352975279897067520

Perhaps this stealth approach was a bit too stealth, because the attention it received was minimal. A couple of friends told me it was cool. I pretty much forgot about it the next day and left it running, figuring it’d be like many of my projects that just toil away for years on their own, chugging along for anyone who happens to stumble across them. But then…

One crazy day

Fast forward about a month. I had just finished up getting a fairly large forearm tattoo the previous night, and I was trying to avoid using my wrist much to aide in the healing (e.g. ideally, avoiding the computer).

Over morning espresso I noticed the source had picked up a few stars on GitHub, which I found interesting, since it had gone fairly unnoticed until that point. Wondering if perhaps someone had mentioned it, I decided to do a quick Twitter search…

Oh shit.

It was certainly out there. Just to be safe I spun up a second web dyno. Within an hour, emojitracker was on the front page of Buzzfeed, Gizmodo, The Verge, HuffPo, Digg… when it happens, it really happens fast, and all at once. Massive amounts of traffic was pouring in.

https://twitter.com/emojitracker/status/360807967232229376

Here’s where Heroku’s architecture really saved me. Although I had never put much initial thought into multiple servers, their platform encourages developing in a service-oriented way that you can naturally scale horizontally. Adding a new web server was as simple as a single command, and it would be up and serving traffic in under a minute, with requests load balanced across all your available instances. Press-drive traffic spikes go away almost as quickly as they arrive, so you’re going to be scaling down as often as you scale up.

Even better, you pay per-minute for web dyno use, which is really helpful for someone on a small budget. I was able to have a massive workforce of 16 web servers during the absolute peaks of launch craziness, but drop it down when demand was lower, saving $$$.

By carefully monitoring and adjusting the amount of web dynos to meet demand, I was able to serve tens of millions of realtime streams in under 24hrs while spending less money than I do on coffee in an average week.

Riding the Wave: Monitoring and Scaling

I primarily used two tools to monitor and scale emojitracker during the initial wave of crazy.

The first was log2viz.

log2viz

Log2viz is a Heroku experiment in which is essentially a simple web visualization that updates with the status of your web dynos based the last 60 seconds of app logs.

I was also periodically piping event data into Graphite for logging purposes.

Graphite charts during launch day.

In order to see total size of pools we want each web streaming server to report independently and have Graphite roll those numbers up. This can be a bit tricky on Heroku because you aren’t going to have useful hostnames but it turns out you can get the short form dyno name by accessing the undocumented $DYNO environment variable, which is automatically set to reflect the current position of the dyno, e.g. web.1, web.2, etc. Thus you can wrap Graphite logging in a simple method:

# configure logging to graphite in production
def graphite_log(metric, count)
if is_production?
sock = UDPSocket.new
sock.send @hostedgraphite_apikey + ".#{metric} #{count}\n", 0, "carbon.hostedgraphite.com", 2003
end
end
# same as above but include heroku dyno hostname
def graphite_dyno_log(metric,count)
dyno = ENV['DYNO'] || 'unknown-host'
metric_name = "#{dyno}.#{metric}"
graphite_log metric_name, count
end

Then you can use the graphite_dyno_log() method to log, and then query in graphite for web.*.stat_name to get an aggregate number back.

Between these two things I was able to have a relatively discrete view of current system status and performance in realtime. You need this sort of realtime data if you’re going to be able to achieve the goal we had here, which was to rapidly scale up and down in response to demand.

I did this manually. That first evening I needed a break from intense computer usage all day, so I actually spent the evening in a bar across the street from my apartment with some friends, having a drink while passively monitoring these charts on some iPhones sitting on the table. Whenever it looked like something was spiking, I used the Nezumi Heroku client to scale up instances from my phone directly. I didn’t even have to put down my drink!

If you have the extra cash, you certainly don’t need to micromanage the instances so closely, just set above what you need and keep an eye on it. But it’s nice to have the option if you’re riding out a spike on a budget.

People have experimented with dyno auto-scaling, but in order to implement it for something like this, you’ll need to have a really good idea of your performance characteristics, so you can set appropriate metrics and rules to control things. Thus, it’s better if you are operating a stable service with historical performance data — it’s not really a realistic option for the pattern of totally obscure -> massively huge suddenly and without any warning.


Things I’d still like to do

There are a few obvious things I’d still love to add to Emojitracker.

Historical Data
This should be relatively simple, I just need to figure out what the storage implications would be and the best way to structure it would be. Showing trend-lines over time could be interesting to see!

Trending Data
Right now the only way to see when things are trending are to eyeball them, but this is a natural thing for Emojitracker to highlight explicitly. This may actually be a prime application for bitly’s ForgetTable project, so hitting up my alma-mater may be the next step.

Alternate Visualizations
Emojitracker does have a JSON API, and the SSE streams don’t require authentication. I’d love to see what more creative folks than myself can come up with for ways to show the data in interesting ways. I’d be happy to work with anyone directly who has a cool idea that requires additional access.

Remember, emojitracker is open source, so if any of the above projects sound interesting to you, I would love collaborators!

Reception and conclusions

Fan Art, via @UnbornOrochi

So was it worth it?

For me creating emojitracker was primarily a learning experience, an opportunity for this non-engineer to explore new technologies and push the boundaries of what I could create in terms of architectural complexity.

Still, it was incredibly gratifying to see all the positive tweets, the funny mentions, and the inexplicable drive of people to try to drive up the standing for poor LEFT LUGGAGE or rally for their latent scatalogical obsessions.

(Since I’ve been told I’m supposed to keep track of press mentions, I’ll post the press list here, mostly so I can ask you all to send me anything that I may have missed!)

The best part has been the people I’ve met through Emojitracker I may not have otherwise. At XOXO one guy came up to me to introduce himself and tell me that he was a big fan of Emojitracker. Suddenly, I realized it was Darius Kazemi (aka @tinysubversions), an amazingly prolific creator of “weird internet stuff” whose work I’d admired for quite some time.

I’ve had the opportunity to work professionally on some amazing things (with amazing people!) in the past. But it was at that point, for the first time, that I felt like that what I had used to consider my “side projects” were now what should define my career moving forward, rather the companies I’d worked at.

I know, and have always known, that Emojitracker is a silly project, one with dubious utility and requiring a bit of lunacy to have spent so much time and effort building. Still, for all the people who saw it and smiled, and may have had a slightly better day than they would have otherwise — it was worth it.

For that reason, I hope that this braindump of how I built Emojitracker will help others to create things that are worth it to them.


ENJOYED THIS ARTICLE?: You might also enjoy the followup post enumerating all the changes involved in scaling over the next 1.5 years here: “How I Kept Building Emojitracker”


Epilogue: Emoji Art Show!

The Eyebeam Art and Technology Center gallery.

I’m thrilled to announce that Emojitracker is going to be featured in the upcoming Emoji Art and Design Show at Eyebeam Art & Technology Center, December 12-14th 2013.

I’m working on an installation version of it now, and there may be a few surprises. Hope to see you there if you are in the New York area!

https://twitter.com/kittehmien/status/405528153151397888

xoxo,
-mroth

P.S. I’m publishing this using Medium as an experiment. If you want to keep up to date with my future projects, the best way is to follow me on Twitter at @mroth.

Next Story — U+1F647 PERSON BOWING DEEPLY 🙇
Currently Reading - U+1F647 PERSON BOWING DEEPLY 🙇

U+1F647 PERSON BOWING DEEPLY 🙇

NOTE: Please see the UPDATE at the bottom of this post! 3/29/16

Nearly three years since they officially blessed it with “partner” access (and after 14 billion emoji tweets tracked), Twitter has decided to shut Emojitracker down.

To be more accurate, they are removing its elevated access to the Streaming API that Emojitracker depends on in order to operate at its high volume:

After a brief investigation, the aforementioned Gnip migration does not seem feasible for me:

  1. When last I evaluated it, the APIs available via Gnip that did not support the level of fine-grained Unicode control Emojitracker currently utilizes.
  2. Gnip costs 💸💸💸.
  3. It is a strong philosophical requirement for me that 100% the work I produce is freely available and open source. The Gnip API is commercial, closed, and proprietary; a normal developer can’t just start hacking on it. So even if Twitter were to give me free access to the Gnip APIs, others then wouldn’t be able to take my code and build upon it the way they have been able to do so previously with Emojitracker. Emojitracker was a wonderful showcase of the possibilities afforded to independent developers (something I spoke a little about in this interview), and I’m sad to see this no longer seems to be important to the Twitter Platform.

Even if for some reason the above were not true—there still would remain the work of migrating a major component of Emojitracker’s infrastructure (which has been performance tuned to somewhat ludicrous degrees over years of development)—with ~30 days notice. To be frank, I’m not interested in investing that level of effort for a very uncertain future on Twitter’s platform. Therefore, my current plan is to shutdown Emojitracker at the end of its partner Twitter Streaming API access on 4/21/2016.

I’ve replied to Twitter asking for an exception, but I honestly don’t expect to receive one—and even if I were to, it wouldn’t help other developers who I believe deserve to get the same level of opportunity as me.

As for Emojitracker, well, it’s had a good run. For a project with a silly premise, it’s been featured in nearly ever single major national newspaper at least once (and many international publications), countless periodicals, even art galleries, and television news. It was the probable data source for 2014’s Word of the Year. It’s even been called out by the Unicode Consortium as a major factor in how new emoji got selected:

It’s also, as those of you who know me well, taken a massive toll on my personal time and ability to work on new projects:

From a technical craft perspective, I learned an immense amount building and scaling Emojitracker— I still believe the best way to push yourself to develop your skill set is to try to build something that seems just on the edge of impossible, but then keep chipping away until it becomes slightly possible. The two rambling blog posts (1, 2) I wrote about my technical misadventures have now been read tens of thousands of times, so I guess there’s some nerd entertainment value contained in there along with my struggles.

So, enjoy it while it lasts! If you want to keep on my newer projects, the best option is to follow me on GitHub where I’m fairly active (or to a much lesser extent: Twitter). I have a number of hopefully fun projects in the works!

👋,
Matthew Rothenberg

P.S. A special thanks to some wonderful people I met through Emojitracker and the emoji community that I might not have had the opportunity to otherwise: Zoë Salditch, Fred Benenson, Jeremy Burge.

FAQs:

What about the Emojitracker API?
I’ll keep it running for as long as it makes sense and I’m able. While the information obviously won’t continue to update past the deadline, I’ll leave it up for people who want access to the historic snapshot from time-of-death. Past that point, my plan is to figure out a reasonable archived format to put that data online in an easily parseable and freely copyable format (thanks Fred for that suggestion!).

What about history trend data from Emojitracker?
Emojitracker was intentionally architected to operate in realtime only, as the immense volume of data it operates on has made storing historic data prohibitively expensive for an independent art project. However, a number of academic papers have been published using samples and subsets of the Emojitracker data. I will try to update this post with links to them soon (I haven’t done a wonderful job of keeping track, sincere apologies for that).


UPDATE 3/29/2016: I’ve reached an agreement with Twitter, and Emojitracker will continue to operate with the elevated access it requires to function. In the interest of full transparency, I’m providing as much followup detail as possible. Read on if you want the extended details, in FAQ format.

What are the terms of my agreement with Twitter?
Emojitracker will continue to have elevated access to the Twitter Streaming API. While I am not allowed to share the specific contract, it is worth noting two key aspects which I can share: a.) a significant factor in our gratis access is based on Emojitracker’s non-commercial status (which is quite fine, as I have absolutely no intention of turning Emojitracker into a business), b.) a provision of the contract is that access must be re-evaluated and renewed on a periodic basis.

In general, is elevated access being discontinued entirely, or not?
I also asked if it will continue to be possible for future independent developers to gain elevated access. They told me that this is possible, but is still the exception rather than the rule. This is something the team there provides support for largely in their free time because they love to see cool stuff. So be respectful of their time and goodwill as they are only able to review and grant elevated access in somewhat rare circumstances.

So why did this happen then?
A number of people asked why I didn’t wait until further reply from Twitter upon receiving the shutdown email (e.g. did I “jump the gun” with the Emojitracker shutdown notification).

Some potentially helpful context is that I already had a long history of communication back and forth with Twitter about Emojitracker, and had undergone a previous formal evaluation regarding the elevated access in October of 2013:

This chain had kicked off a month-long process of dozens of emails back and forth, with master lists of contact information, API keys, etc. being put on file.

So when I received this new email (with quite different wording), it read to me as an unequivocal policy decision that had been made about the future of the elevated access to the streaming API for all users. As a result, I started to make preparations to shutdown Emojitracker and inform my user base on a tight timeframe. ( it certainly didn’t help that I received the email on my way to the airport for an extended trip that would cover much of the 30 day shutdown window, and thus spent most of the flight writing a stressed-out response and trying to get everything handled care of as quickly as possible.)

It’s unfortunate that so much misunderstanding could happen when I have had so much communication with Twitter over the past few years about Emojitracker. However I believe I failed to realize the extent to which Twitter is a big organization now, and many of the people I’ve dealt with in the past simply don’t work there anymore. Even though the information was supposedly “on file,” two years of organizational entropy can result in a situation where that information can be effectively nonexistent for the people who are currently trying to do their jobs.

If nothing else, the experience has reminded me how grateful I am to be working at an organization that embraces radical email transparency.

How do I feel now about the Twitter and the Twitter Streaming API?
The Twitter Streaming API remains one of the best APIs I’ve ever worked with. It has some quirks, but even with normal non-elevated access it offers unparalleled levels of power to an independent developer. It really is in a league of it’s own. When you compare it to something like Instagram’s “Realtime” API, it’s just no contest. It would be entirely impossible to do anything like Emojitracker on any other major platform out there today (and believe me, people have asked me to).

While I might not agree with every business decision Twitter the corporation makes, I want to explicitly call out that the people who work there deserve respect. They are doing a difficult job, and have to weigh numerous concerns. While I was clearly disappointed when I thought elevated streaming API access was getting shut down, and I’m sure that disappointment came through in my writing, some of the responses I saw on Twitter and Hacker News bordered on being personal attacks against specific staff members at Twitter, which makes me deeply uncomfortable. I cannot state this strongly enough: Open source developer community, if we want to be treated with respect, then we need to treat others better ourselves.

Does this mean Emojitracker is here to stay?
I’ll be frank. The biggest threat to Emojitracker’s continued existence is not (and never was) Twitter. While I’ve put years of work into Emojitracker, I have many other more projects I would prefer to spend my time on these days, not to mention a full-time job. Emojitracker needs an immense amount of work to keep running efficiently and stay up to date. And it needs help.

  1. If you are a software developer, Emojitracker needs more community involvement. There is no doubt it’s a complex project, and a little daunting to get involved in at first. I would like to provide more mentorship to someone who wants to take something like that on — if there is enough interest (let me know!), I will host an online Google Hangout in a few weeks to help onboard people with contributing code.
  2. Right now Emojitracker is paid for 100% out of pocket by me. While I have received some donations, the full amount of donations over the ~3 years of the project would cover about one month of hosting. If Emojitracker is upgraded to support newer emoji, those costs will go up. If you want to see it stick around, please donate and help cover the cost of hosting. Or if you work for Heroku, and are interested in sponsoring our hosting, I’d love to talk to you.
  3. In my fantasy world, I’d raise enough money to pay an intern a fair wage to hack on Emojitracker as well. (Note I don’t need to make any money myself, but I am uncomfortable with exploiting the labor of others…) It’s too late to apply for Google Summer of Code now, but this is something I’ plan to look into for the future. Any similar arrangements people can recommend would be appreciated!
Next Story — WTF Just Happened at Khan Academy
Currently Reading - WTF Just Happened at Khan Academy

WTF Just Happened at Khan Academy

TLDR: we have a new team called “WTF” focused on what might be termed meta-engineering: expertise in the practices, tools, and culture of a modern-day software team.

When I first came to Khan Academy last year, I discovered one of the healthiest engineering cultures of any software team I’d ever seen. Just ask the handy Slack Bot:

Khan Academy’s Culture Cow Slack bot dispenses wisdom.

Most importantly, core to KA’s dev culture is the notion that “anyone can fix anything.”

In such an environment, starting an entirely new functional team within the engineering organization to focus inward might seem strange—Our engineering culture and practices were already pretty darn good. However, we asked ourselves: What if pretty-darn good wasn’t good enough? What if our goal was to make Khan Academy the absolute best place in the world to be a software developer? What would we do then?

So WTF is it then?

WTF is focused on how to build software as a team — everything from process management, to tooling, to making sure engineering best practices are well defined yet evolving, to improving how engineers & designers interface with the rest of the company.

Members of this team are software engineers who work on the same project list and teams as everyone else. We felt it was critically important to not create a separate service organization, and since KA already has a culture where any developer is allowed (and expected) to implement these sort of changes, people jump between different types of projects all the time.

Rather, just like some engineers choose to focus on front-end or mobile or infrastructure as their primary area of expertise (while still working across the stack as needed), we wanted to recognize another growing functional area of focus. One might call it “meta-engineering” — expertise in the practices, tools, and culture of a modern-day software team.

So while members of the team work on all sorts of projects, they are expected to use their time embedded in those projects to develop this expertise: troubleshoot process, introduce and trial new ways of working, and develop best practices. The commitment from management to someone on this team is to explicitly help provide mentorship and career development opportunities along those lines.

The Hardest Problem in Computer Science is Naming Things

In defining this team, we were still having a particularly difficult time with one particular aspect — seemingly the most basic part: WTF to call it.

Many organizations have a “Internal Tools” team, but we knew that wasn’t right for us. Internal tools typically means “building things used internally outside of the core product,” whereas we want to be focused on “how the team that building the core product works.” (Also, at far too many organizations, this unfortunately becomes the unoffical “dump all the unglamorous projects no one wants to touch here” department, and we certainly want to avoid that stigma.)

Twitter has the notion of “Engineering Effectiveness,” which sounded closer on paper, and certainly more broadly defined. But I was stuck with a few fundamental differences with the way it was discussed and some of our aims:

  • We care just as much about developer happiness as we do organizational effectiveness.
  • We want to address cultural issues, not just technical ones.
  • Don’t become the Department of Saying No™. One of the key guidelines we established early on in thinking about this was “it must enable and push us to say yes to more.”

In terms of prior art, the spirit we want to embody is most closely aligned with Ernie Miller’s concept of “Humane Development,” which focused on taking a holistic and human-centric approach to software development.

But what do you call a functional team around pushing those concepts forward?

When discussing what we wanted this team to be, a particular anecdote kept coming up about the process by which “suboptimal” things rarely get fixed (as told by Julia Evans to Dan Luu):

new person joins
new person: WTF WTF WTF WTF WTF
old hands: yeah we know we’re concerned about it
new person: WTF WTF wTF wtf wtf w…
new person gets used to it
new person #2 joins
new person #2: WTF WTF WTF WTF
new person: yeah we know. we’re concerned about it.

At one point, in exasperation, I said “why not just call it WTF?” as a joke, and it kinda stuck. We quickly came up with something that fit that acronym (“Workflow Task Force,” I believe, but that varies based on who you’re talking to at the moment), but the reality is the name was simply meant to invoke the spirit of this anecdote.

Things I should have learned from classic Simpsons episodes: “His name isn’t important!”

…But yeah, there is actually a box on the org chart now that literally says WTF on it. Whoops. My bad. Maybe we’ll change it later. Maybe not. But it seems to be working for now!

The primary point of writing this post was to share our efforts in thinking towards this, so that we can begin to engage in public dialog with others in the industry about their experiences. If your organization is working on something similar, I’d love to talk.

Some early goals

Focus on the holistic experience of the developer, not just tools. The experience of being a productive and happy software developer on a team is defined by much more than the tools one uses. (As an example, the first few initiatives WTF team members have been working on have ranged from scaling how we track bug reports to containerization of our dev environment to encouraging diversity through our hiring practices.)

Enable thoughtful change. We never want to be too comfortable, or become complacent with the current state of things. On the other hand, we value thoughtful approaches to problems, and are extremely skeptical of the stereotypical Silicon Valley “disrupt all the things for the sake of disruption!” approach. Focus on thoughtful solutions to benefit the entire team (including those who haven’t joined yet), and go beyond existing best practices when possible. In short, build the industry we want to work in.

Learn from remote work. Khan Academy currently has 17 (edit: now 18) full-time remote employees (while not a majority, a significant portion of our small team) — but more importantly, the growing focus on remote work is teaching all of us a ton about more effective and healthier ways for us all to work.

The Obligatory “Yeah, We’re Hiring”

Well, if I’m going to make this post public I’d be remiss not to mention this as well. So uhm, in my unofficial wording: We value people who are skilled yet humble, as excited to teach as they are to learn, and believe in the promise of technology to make the world a better place for more than just a small set of privileged individuals. (Also: we may be a non-profit, but we compensate fairly.)

If you think this might be you and you are particularly interested in WTF, please reach out to me directly.

Next Story — How I Kept Building Emojitracker
Currently Reading - How I Kept Building Emojitracker

How I Kept Building Emojitracker

Photo credit: WALL STREET JOURNAL / Keith Bedford.

The previous post in this series, How I Built Emojitracker, was quite popular despite being long and rambling, and continues to be linked to fairly heavily. However, quite a bit has changed since then, so 1.5 years later, it seems an update about Emojitracker was overdue. (P.S. If you haven’t read that one, this one won’t make any sense, so do that first.)

Traffic and press coverage have continued relatively unabated, at this point I’m reasonably confident the project is here to stay. Emojitracker has now analyzed 8.9 billion tweets, and the press continues. Emojitracker was even used to determine the choice of ❤️ as the word of the year for 2014, and then there was this bombshell, confirming Emojitracker’s acceptance into the Emoji Illuminati:

While I will only focus on the major technical updates in this article, I feel I should note that a fair amount of nontechnical work goes into the Emojitracker “empire.” I wish there existed a better open-source contribution model for thing such as marketing, community management, press relations, etc, because there is a ton to be done to keep this thing going that could benefit from the help of not just technical people. If someone knows how to make this happen, I’d love to hear your ideas.

But this post is going to be highly technical. Worth caveating, while the previous post had a narrative arc that tried to explain things to newcomers, this one is going to be mostly a brain dump of pure technical updates due to time constraints — so that people relying on the previous as a technical guide to Emoji or Emojitracker’s architecture are not getting out of date information. I started the last post with the caveat “I am not an engineer” (which I caught a bit of flack for)— I’ll start this one with the caveat “I am not a writer.” Here be dragons! (🐲🐲🐲)

In this post, my plan is to discuss:

  • The modularization of Emojitracker into over a dozen independent open source projects.
  • Updates to UTF encoding of Emoji (“variant encoding”) that offered a new solution to one of the frontend rendering issues discussed in the previous post.
  • The optimization process of the emojitrack-streamer service — including standardizing the API, building an acceptance framework for that API, building benchmarking tools, and then finally doing a platform migration to observe the results — twice. (The end result being a 25x increase in concurrent streaming connections per server.)
  • The optimization for the emojitrack-feeder service — lots of twiddly micro-optimizations involving memory allocation, event loop handling, and hopes and dreams.

Modularization of services

As Emojitracker has grown, it’s become necessary and helpful to break out components of it into independent projects, which can be tracked and updated independently, with different dependency chains.

To see this idea at a glance, below are the Emojitracker repositories on GitHub, with activity visualized over time:

GitHub repos for the Emojitracker project over time.

While “service oriented architecture” is increasingly becoming a meaningless buzzword, it does pretty accurately describe the Emojitracker platform today. Components communicate strictly via defined APIs over the network.

The key advantage of this modularization has been it has become significantly easier to upgrade features for certain components of the project in isolation , or even perform a number of platform migrations completely invisibly to the other components of the overall website.

One of the tricky things I’ve noticed is that when a project becomes complex enough to span so many separate repos is it’s hard to get new contributors up to speed. I’ve turned the main Emojitracker repo into a “table of contents” of sorts, but I’d love to figure out how to do more here.

Unicode Variation Selectors

In the previous post, I discussed the problem stemming from trying to render emoji characters that also had a Unicode plaintext equivalent. If you recall, we relied upon some pretty nasty font-face hacks to work around the issue.

Thankfully, with Unicode 6.3 implementation becoming more adopted, there is a much cleaner solution. Unicode specifies what are known as “variation selector” code points, which can come after a codepoint and are used to indicate an alternate variation of the glyph. There are two in particular that we care about for this.

The example from the previous blog post, now handled via Unicode variation selectors.

VS15 (U+FE0E) specifies the “text” variation of the glyph, whereas VS16 (U+FE0F) will get you the Emoji bitmap glyph. Support for this has been built in to MacOSX and iOS for a while now, so Emojitracker has been switched over to using that mechanism, eliminating the complex hacks from before. Instead, we simply rewrite all Emoji that support variants (but don’t have one specified) to use the optional variant form. This means more multiple-codepoint characters, but we were already dealing with that anyhow.

The positioning of the variant selector is not necessarily where you would expect it.

Of course, as you probably already know from this series of posts, it’s never actually that simple. There are some gotchas to be aware of: for example, as seen above, in some of the existing double-codepoint Emojis, the variation selector actually goes in the middle. Wat. Oh well.

I talked about this in more detail in my BrooklynJS talk (naturally titled “👨💬 💻🎨📈”), but the simple version is there are some complex and seemingly arbitrary rules that determine all this. Your best bet is really to have a library that just has all the edge cases hard coded in already. Naturally, this is now supported in my EmojiData library out of necessity (as well as its Javascript and Erlang/Elixir ports).


Optimizing for High Volume Streams


The most heavy portion of Emojitracker are the web streamer boxes, which consume the internal Pubsub event feed from Redis and push out relevant events to connected clients via SSE.

This is the part of Emojitracker that needs to scale with traffic — in the previous post, when I mentioned spinning up extra servers during usage spikes, these were the types of boxes we need more of.

There are a lot of websocket/SSE libraries out there that claim to be “high performance,” but their numbers typically assume a published event rate on the order of a chat app broadcasting out messages a few times a second. Emojitracker streaming servers need to send a minimum of 60 messages per second to every single client (more if they are subscribed to detail streams as well), so it’s considerly more demanding. The original production version I wrote in Ruby could roughly handle 40–50 simultaneous clients on a server instance before it started to see some latency degradation.

That number is actually not terrible, but as people tend to leave Emojitracker up and running in a window (sometimes for hours or even days), usage spikes related to major press coverage would sometimes mean I could need a few dozen streamer servers running to handle the load — maybe not terrible for a business, but for me this was all paid for out of pocket. With the popularity of Emojitracker still continuing to grow, I needed a more sustainable solution.

The ideal thing to do first in these situations is to try to rely on something off-the-shelf, hopefully someone else has solved the problem for you. I tested a number of the popular existing libaries with mock data to see if they could do better, and found that my results were, unfortunately, pretty consistent with everything else out there. But could this really be the best possible? After doing some back-of-envelope math, it seemed like there was no theoretical reason that the hardware shouldn’t be able to support at least an order of magnitude more capacity.

In order to do better, I was going to have to roll my own.

Developing a Streamer Spec

Before embarking on a port, I knew I was going to have to be careful to make sure the new version handled everything in the API exactly the same way, so that I could seamlessly swap it in. This was a critical part of Emojitracker’s infrastructure, and replacing it would be a risky operation.

The best way to document is to document in code, so I created an acceptance framework that could be run directly against a staging server and make sure it did everything properly. This is a situation where unit testing is not the best solution, as I wanted to verify behavior cross-platform, and in environments where other factors (the routing layer, for example) would have impact on the results.

This proved to be invaluable, and additionally helped nail down some of the “taken for granted” particulars of the API in more explicit detail than before. (I also owe a debt of gratitude to the excellent Heroku routing team who spent a great deal of time and energy to help debug some of the more arcane interactions experienced at the routing layer.)

Benchmarking Tools

Of course, even if the port worked, I still needed to know “is it faster?” Turns out this is actually a nontrivial question to answer when it comes to HTTP streaming. Most HTTP benchmarking tools try to open as many connections as possible, primarily in serial, and measure how many connections can be completed (e.g. closed) in a time period.

SSE is an entirely different ballgame despite occuring over normal HTTP connections. I knew from previous monitoring that what I needed to monitor was the actual message received rate spread out across multiple connected clients, versus the expected rate of sent messages. Additionally, there were cross-interactions between the number of existing streams and their message receive rate and the latency of new stream subscriptions. No existing benchmarking tools monitored the proper dynamics for this situation.

So in order to do this, I again had to roll my own (sigh). I reluctantly created a benchmarking tool sse-bench, which tests exactly that. It still needs some cleanup but it was mostly serviceable enough for the task at hand.

Enabling Production Testing

Testing in isolation isn’t enough — the real world is messy, so to avoid unexpected results, you need to be able to test in production, with real traffic. …But of course, you also don’t want to break everything.

The emojitrack-web frontend was modified so that based on configuration variables, a certain percentage of clients could be directed to different streaming servers. This allowed me to test alternate implementations of the streaming server with real traffic by handing them some small percentage of production traffic and seeing how they handled the load.

Porting to NodeJS

The ideal candidate for the job (to my naive mind) seemed to be NodeJS. After all, it excels at I/O, and this was I/O, right?

Using the streamer-spec, it was easy to write a complete port to NodeJS, which ended up feeling pretty clean to me (using CoffeeScript made it easy to port the Ruby code, since they use similar idioms). But when I benchmarked it, I was disappointed. It was better, but not significantly better. Depending on the situation, it seemed to buy about an overall 20% increase in capacity, and also gained a more linear degradation curve once it went over capacity. Still, being able to handle usage spikes with 10 servers instead of 12 didn’t seem like a massive win to me, and my pencil calculations suggested there was still a lot of potential remaining. This is the future, I was promised jetpacks!

I’d still like to know why it didn’t perform better, but at the time I asked 5–6 people who knew NodeJS well to review it and tell me what was wrong, and each of them came up empty handed in terms of yielding big perf changes. I still think someone smart can find the answer and make it super fast, and I’m sure that now that I’m publishing this someone will and embarrass me (please do!). But at the time, I got frustrated and moved on. Performance shouldn’t need to be a black art with arcane rituals and secret knowledge. I needed something better suited for my use case.

Porting to Go

Finally, I tried porting it to Go. Go’s concurrency primitives were actually extremely well suited to the data pipeline here, and it ended being mostly quite fun to write (albeit requiring an extremely different way of thinking about the pipeline, and significantly more code).

After a little tuning, the results were surprising: I was able to support about ~1200 simultaneous clients per server before seeing latency degradation, a 25x improvement. This means that in almost all cases, 100% of Emojitracker’s stream traffic, even during usage spikes, can be handled with only two streaming server instances. Phew!

(I carved out the generic portion into a library that be used for your own streaming needs, found on GitHub as sseserver. In my casual benchmarking, it handles streaming 100,000 messages per second to web clients without breaking a sweat. Share and enjoy.)

When the time came to bring in this replacement for real, the aforementioned A/B infrastructure enabled me to ramp it in slowly, progressively moving the traffic balance over time. For the first day it handled 20% of all production traffic, then for a week I sent 50% to each platform, comparing how they handled various production situations and tuning. Finally, full cutover. All completely invisible to the web clients. And I could afford to buy lunch at a New York restaurant again (well, almost).


Feeder Optimization


While the streamers bear the bulk of Emojitracker’s scaling needs, the feeder is of utmost importance, and the pressure on it had been increasing steadily as the global popularity of emoji grew.

Emojitrack-feeder does three primary things:

  • Streams from Twitter Streaming API: handles connections, reconnects and backoffs, and event/error handling for streamed items from the API.
  • Process entities: deserialize JSON, pattern match on content to identify Emojis, pick from and transmute a number of fields to build our optimized output data.
  • Push to Redis: Updates a number of keys for each emoji symbol, and pushes to PUBSUB streams.

Relatively simple, right? But as we’ve learned: at very high volume, nothing is simple.

Stream processing of this kind can be notoriously difficult to parallelize. Since we only get the source from Twitter as a single stream, we’d have to manually fan out to distributed systems on our side, and the overhead of doing this communication is actually far greater than the gains from parallelizing in our configurations, since we are significantly more IO bound than CPU bound (e.g. we already spend more time moving things around than performing operations on them). So keeping things on a single system is better in our case.

At a current average of ~500 tweets/second, this means the total time to fully process each tweet has to be under 2 milliseconds. Peaks and bursts push that significantly lower. So to scale emojitrack-feeder to match expected growth of Emoji usage worldwide, I had to get into some serious optimizations. Let’s talk about some of the more interesting ones.

Process entities: Avoiding unnecessary type conversion and memory allocation

The Ruby gems tend to deserialize JSON into Twitter::Tweet objects which are highly optimized, and nice for using in Ruby. Some things are almost too nice. For example, if you access a field that represents a specific data type, you get back a native Ruby class (DATE or URI, for example), instead of a string.

Now, the Twitter gem is very smart about this, and only does this conversion upon first demand (via method name overloading), and then memoizes the results so subsequent access is fast.

However, in our case, we’re going to be translating those URLs right back into strings anyhow, so we’re doing a back and forth double type conversion that’s unneeded. What’s worse, the memoization is not needed by us since we only access the fields once, but it creates a heap memory allocation which later has to be garbage collected. Why does this matter? Bear in mind, in our use case we create and then discard thousands of these objects per second.

To get around this, I modified our code to sneak past the accessor methods and peek at the attribute properties directly, use them directly when creating our new ensmallened data object, which cut out most of the unnecessary conversions/allocations and sped up that hot path.

ALWAYS BENCHMARK these sort of changes. In this case I used the benchmark-ips library to compare the performance of the relevant methods with every single micro-change, before and after, so I could see what made a difference of how much. In one iteration of my improvements I was able to pretty much destroy all performance gains with a tiny change, which I would have never noticed if every part of the overall change wasn’t comparatively benchmarked, and rather I only looked at the changeset as a whole. Real world applications are complex and can defy all common sense when it comes to performance—always measure when it matters.

Pushing to Redis: Move pipeline creation to the server

If you recall from the previous blog post, our Redis writes were fairly simple:

For each “update”, we were actually making several distinct Redis queries. To make it more efficient, they were pipelined such that they are all sent in bulk, avoiding back and forth round-trip time over the network.

However, I realized there was some necessary duplication with this approach — some of those queries contained the same data, so the output network IO (in terms of bandwidth) was larger than it needed to be. Additionally, our client still has to parse the success status of all those commands, which adds a tiny bit of overhead.

Interestingly enough, Redis actually allows for server side execution of scripts. So what we can do instead is push as much of the logic to a Lua script which Redis can execute natively, and perform a remote-procedure-call from our script. So instead, we use the following Lua script:

When emojitrack-feeder boots, this script is loaded directly onto the Redis server via the SCRIPT LOAD command, which returns a SHA1 digest to reference the script. We then can use that digest to make a single Redis call via EVALSHA and pass along just our two variables — everything else then happens server side. The new code execution on our side therefore simply looks like this:

REDIS.evalsha(sha, [], [matched_emoji.unified, status.tiny_json])

That’s it! By reducing down to a single remote call, this not only cuts down on bandwidth and feeder processing requirements, it opens it up to more easily adding additional database side functionality per each update in the future, moving responsibility out of the feeder app itself.

Longer term: migration to another platform?

TLDR: Probably not.

I started with looking at a full port to a different language, after all “Ruby is slow lol.” and as seen in the previous section, I’d had great success with the Go port of emojitrack-streamer.

However, never blindly trust prevailing wisdom. The maturity of the libraries you are going to use and the methods you use them with will make a lot more difference than the platform itself.

In my case, emojitrack-feeder is highly dependent on Twitter Streaming API and Redis libraries. There are thankfully good off-the-shelf Redis libraries out there for most platforms, but Twitter Streaming API libraries are another story.

The twitter and tweetstream gems for Ruby are excellent — especially when compared to what exist on other platforms. Most other implementations lack proper error/event handling, and I found even the “fast lol” languages such as NodeJS and Go had libraries with performance characteristics that were lacking in comparison. I have a very detailed, ongoing investigation of this with benchmarks that you can read about here: the twitter streaming showdown.

Current library benchmarks aside, if I ever do decide to do a platform migration, Elixir is the most promising bet, as it offers some capabilities that would be very labor-intensive to replicate on other platforms. Being built on the Erlang VM (BEAM) means amazing support for distributed message passing, supervision trees, and even code hot-swapping, where a module can be updated in place without restarting the app.

To aide in future experiments, EmojiData library has been ported to both Elixir/Erlang and Javascript, with very promising performance results (especially the Elixir port which relies on binary pattern matching instead of regular expressions for scanning text — but that’s the subject for another blog post!).


Future Plans

Well, perhaps most pressingly, there’s now this:

With these newly standardized Fitzpatrick Skin Type Emoji Modifiers, there’s still a ton of work to be done just to stay up to date with the evolving Emoji landscape. Think that one is going to be easy? If you’ve been paying attention to anything in these articles thus far, you can guess that’s pretty unlikely. So, if you’d like to get involved in helping make Emojitracker more diverse (there should be some interesting data with this one!), ping me.

As Emojitracker now approaches 10 billion tweets I know the scaling work will also have to continue. These Emoji thingamabobs aren’t going away anytime soon, and the rate continues to rise.

As for everything from first article I wanted to do — history over time, predictions, trend detection — nope! Just scaling the service has been pretty much my full-time side-project job. I’d love for some smart people to take this thing further than I can on my own. Maybe you are a data science student looking for a huge realtime dataset to analyze? I have millions of smiling poops for you. Perhaps you’re interested in making dynamic visualizations to do something interesting with hundreds of emoji tweets per second? The streaming API is public, and I’d love to help you get started.

If you’d like to contribute with code (or just hang out), you can now find me in #emojitracker on Freenode IRC. If you aren’t an open-source developer or don’t have time, you can also help out by donating via Square Cash or Gratipay to help pay the server bills.

Thanks everyone for the continuing wild ride.


Want to keep up with the other things I’m building? Follow me on Twitter and/or GitHub.

Next Story — Swipe Left
Currently Reading - Swipe Left

Swipe Left

Dating Apps and Drone Strikes


NOTEBOOK: For the past few months I have been experimenting with inserting simulated drone strike “kill decision” imagery into the Tinder dating app. I’ve been trying out various things, ranging from aerial drone photography to mugshots of targets of previous drone strikes in the Middle East and South East Asia.

Screenshots from various experiments: “Haki” (Hakimullah Mehsud) Tinder profile; drone strike imagery in the new Tinder “moments” feature; and responses.

I don’t have a grand game plan for this project yet, but sometimes I find value in exploring an idea “live.”

So, some rambling notes from my files to document the “why” I ended up doing this below, in particular what parts interested me. (Note: I worked on and wrote this prior to the recent revelations about the unacceptable executive behavior within Tinder (the company), and haven’t had time to digest how that impacts the way I had been thinking about this project and whether I want to continue working with their platform. So for now, I’m just going to publish this as-is, and perhaps come back to it later.)


A large part of the popularity of Tinder is owed to the user interface. On a high level, Tinder essentially uses UI to make what should be difficult (picking a potential partner) into a simple game.

The interface of Tinder is consciously reductionist. You get a name, age, and (sometimes) a very brief bio. The decision tree is binary: yes or no (or in Tinder UI, swipe right or left). No winks, nudges, or ratings. No bookmarking to come back later for decision. You have to make a decision in order to move on.

Perhaps most interesting is how all actions in Tinder are final. Accidentally swipe left on someone who you thought might have actually been interesting? Too late, they’re gone forever. To combat loss-aversion, Tinder aggressively attempts to de-emphasize the significance this interaction — more and more people will come, so just make a decision quickly and move on. The addictive magic of Tinder lies in an interface that pushes constant momentum. No time to feel bad. Reflection and contemplation would break the magic cycle.

(To be fair, this magic cycle is part of what makes Tinder work so well for it’s intended purpose. No one wants to participate in an online dating site that makes them feel worse.)

The nature of computer interface to modify how we feel about “difficult” activities is intriguing to me. User Experience Design is often about trying to make complex actions achievable, but to me this feels like something different — using interface to make us feel “okay” about the reduction of complex activities to simple ones. What the full consequences are of reducing these complex activities to simple actions is unclear.


So let’s talk about drone strikes.

Part of the reason drone strikes produce such a visceral response in people is the sense of dehumanized technology — killer robots that kill without judgement or conscience based on algorithmic criteria. Reports such as this one by the Human Rights Watch, all echo the importance that a human being, instead of an algorithm, be a gating factor in the approval loop of making the final kill decision.

Active ordinance drone missions are typically carried out via a secure compound located 45 miles outside of Las Vegas. While the piloting of drones may take place in suburban Nevada, kill decisions frequently appear to go all the way to the top.

But what about the interface these decisions are carried out via? What does it really mean to be “in the loop” in a computer mediated interface?

Here are some select phrases from a well circulated New York Times article about drone strikes, focusing on the experience of President Obama reviewing the kill list:

“The mug shots and brief biographies resembled a high school yearbook layout…[President Obama] took a moment to study the faces…‘How old are these people?’ he asked.”

A list consisting of members who met a series of criteria.

Photo, age, a brief bio.

A binary decision.


The similarities between these tasks struck me.

I wondered what the computer mediated interface for reviewing drone strike targets might look like, and wanted to attempt to simulate the reduction of complex judgements with consequences into simple binary UI decisions.

Which is how I arrived at my little experiment, and for a period of time some New Yorkers would find drone strikes images interspersed with their continuous yes-and-no swiping to Tinder matches of shirtless-ab-photos and Instagram-filtered-art-selfies taken in the MoMA rain room, forcing them to make just one more binary decision before moving on.


I generally attempt to produce work that is apolitical in nature. Thus I feel it’s important to note that my interest in exploring this project is not in exploring foreign policy (although I suppose some comparisons are inevitable). Rather, the main intellectual question which has driven my interest in this topic is the ability of computer mediated user-interfaces to produce a distancing barrier between a viewer and an action that aides in creating quick action. I’m curious about the effectiveness of UI in “helping” make the difficult less difficult, and whether the consequences of doing so are always desirable.

As a side note: Tinder turns out to be a great platform for performance art. Especially interesting to me is that Tinder members who encounter the images are forced to make a decision. In the UI of Tinder, they are simply not allowed to move on until they’ve done so. (For everyone who asks “how many people swipe left?”—this really more about performance art than data viz in concept. How viewers choose to react is far less interesting to me than the fact that they’ve been forced into this situation to begin with.)

Sign up to continue reading what matters most to you

Great stories deserve a great audience

Continue reading