U+1F647 PERSON BOWING DEEPLY 🙇

NOTE: Please see the UPDATE at the bottom of this post! 3/29/16

Nearly three years since they officially blessed it with “partner” access (and after 14 billion emoji tweets tracked), Twitter has decided to shut Emojitracker down.

To be more accurate, they are removing its elevated access to the Streaming API that Emojitracker depends on in order to operate at its high volume:

After a brief investigation, the aforementioned Gnip migration does not seem feasible for me:

1. When last I evaluated it, the APIs available via Gnip that did not support the level of fine-grained Unicode control Emojitracker currently utilizes.
2. Gnip costs 💸💸💸.
3. It is a strong philosophical requirement for me that 100% the work I produce is freely available and open source. The Gnip API is commercial, closed, and proprietary; a normal developer can’t just start hacking on it. So even if Twitter were to give me free access to the Gnip APIs, others then wouldn’t be able to take my code and build upon it the way they have been able to do so previously with Emojitracker. Emojitracker was a wonderful showcase of the possibilities afforded to independent developers (something I spoke a little about in this interview), and I’m sad to see this no longer seems to be important to the Twitter Platform.

Even if for some reason the above were not true—there still would remain the work of migrating a major component of Emojitracker’s infrastructure (which has been performance tuned to somewhat ludicrous degrees over years of development)—with ~30 days notice. To be frank, I’m not interested in investing that level of effort for a very uncertain future on Twitter’s platform. Therefore, my current plan is to shutdown Emojitracker at the end of its partner Twitter Streaming API access on 4/21/2016.

I’ve replied to Twitter asking for an exception, but I honestly don’t expect to receive one—and even if I were to, it wouldn’t help other developers who I believe deserve to get the same level of opportunity as me.

As for Emojitracker, well, it’s had a good run. For a project with a silly premise, it’s been featured in nearly ever single major national newspaper at least once (and many international publications), countless periodicals, even art galleries, and television news. It was the probable data source for 2014’s Word of the Year. It’s even been called out by the Unicode Consortium as a major factor in how new emoji got selected:

It’s also, as those of you who know me well, taken a massive toll on my personal time and ability to work on new projects:

From a technical craft perspective, I learned an immense amount building and scaling Emojitracker— I still believe the best way to push yourself to develop your skill set is to try to build something that seems just on the edge of impossible, but then keep chipping away until it becomes slightly possible. The two rambling blog posts (1, 2) I wrote about my technical misadventures have now been read tens of thousands of times, so I guess there’s some nerd entertainment value contained in there along with my struggles.

So, enjoy it while it lasts! If you want to keep on my newer projects, the best option is to follow me on GitHub where I’m fairly active (or to a much lesser extent: Twitter). I have a number of hopefully fun projects in the works!

👋,
Matthew Rothenberg

P.S. A special thanks to some wonderful people I met through Emojitracker and the emoji community that I might not have had the opportunity to otherwise: Zoë Salditch, Fred Benenson, Jeremy Burge.

FAQs:

What about the Emojitracker API?
I’ll keep it running for as long as it makes sense and I’m able. While the information obviously won’t continue to update past the deadline, I’ll leave it up for people who want access to the historic snapshot from time-of-death. Past that point, my plan is to figure out a reasonable archived format to put that data online in an easily parseable and freely copyable format (thanks Fred for that suggestion!).

What about history trend data from Emojitracker?
Emojitracker was intentionally architected to operate in realtime only, as the immense volume of data it operates on has made storing historic data prohibitively expensive for an independent art project. However, a number of academic papers have been published using samples and subsets of the Emojitracker data. I will try to update this post with links to them soon (I haven’t done a wonderful job of keeping track, sincere apologies for that).

UPDATE 3/29/2016: I’ve reached an agreement with Twitter, and Emojitracker will continue to operate with the elevated access it requires to function. In the interest of full transparency, I’m providing as much followup detail as possible. Read on if you want the extended details, in FAQ format.

What are the terms of my agreement with Twitter?
Emojitracker will continue to have elevated access to the Twitter Streaming API. While I am not allowed to share the specific contract, it is worth noting two key aspects which I can share: a.) a significant factor in our gratis access is based on Emojitracker’s non-commercial status (which is quite fine, as I have absolutely no intention of turning Emojitracker into a business), b.) a provision of the contract is that access must be re-evaluated and renewed on a periodic basis.

In general, is elevated access being discontinued entirely, or not?
I also asked if it will continue to be possible for future independent developers to gain elevated access. They told me that this is possible, but is still the exception rather than the rule. This is something the team there provides support for largely in their free time because they love to see cool stuff. So be respectful of their time and goodwill as they are only able to review and grant elevated access in somewhat rare circumstances.

So why did this happen then?
A number of people asked why I didn’t wait until further reply from Twitter upon receiving the shutdown email (e.g. did I “jump the gun” with the Emojitracker shutdown notification).

Some potentially helpful context is that I already had a long history of communication back and forth with Twitter about Emojitracker, and had undergone a previous formal evaluation regarding the elevated access in October of 2013:

This chain had kicked off a month-long process of dozens of emails back and forth, with master lists of contact information, API keys, etc. being put on file.

So when I received this new email (with quite different wording), it read to me as an unequivocal policy decision that had been made about the future of the elevated access to the streaming API for all users. As a result, I started to make preparations to shutdown Emojitracker and inform my user base on a tight timeframe. ( it certainly didn’t help that I received the email on my way to the airport for an extended trip that would cover much of the 30 day shutdown window, and thus spent most of the flight writing a stressed-out response and trying to get everything handled care of as quickly as possible.)

It’s unfortunate that so much misunderstanding could happen when I have had so much communication with Twitter over the past few years about Emojitracker. However I believe I failed to realize the extent to which Twitter is a big organization now, and many of the people I’ve dealt with in the past simply don’t work there anymore. Even though the information was supposedly “on file,” two years of organizational entropy can result in a situation where that information can be effectively nonexistent for the people who are currently trying to do their jobs.

If nothing else, the experience has reminded me how grateful I am to be working at an organization that embraces radical email transparency.

How do I feel now about the Twitter and the Twitter Streaming API?
The Twitter Streaming API remains one of the best APIs I’ve ever worked with. It has some quirks, but even with normal non-elevated access it offers unparalleled levels of power to an independent developer. It really is in a league of it’s own. When you compare it to something like Instagram’s “Realtime” API, it’s just no contest. It would be entirely impossible to do anything like Emojitracker on any other major platform out there today (and believe me, people have asked me to).

While I might not agree with every business decision Twitter the corporation makes, I want to explicitly call out that the people who work there deserve respect. They are doing a difficult job, and have to weigh numerous concerns. While I was clearly disappointed when I thought elevated streaming API access was getting shut down, and I’m sure that disappointment came through in my writing, some of the responses I saw on Twitter and Hacker News bordered on being personal attacks against specific staff members at Twitter, which makes me deeply uncomfortable. I cannot state this strongly enough: Open source developer community, if we want to be treated with respect, then we need to treat others better ourselves.

Does this mean Emojitracker is here to stay?
I’ll be frank. The biggest threat to Emojitracker’s continued existence is not (and never was) Twitter. While I’ve put years of work into Emojitracker, I have many other more projects I would prefer to spend my time on these days, not to mention a full-time job. Emojitracker needs an immense amount of work to keep running efficiently and stay up to date. And it needs help.

1. If you are a software developer, Emojitracker needs more community involvement. There is no doubt it’s a complex project, and a little daunting to get involved in at first. I would like to provide more mentorship to someone who wants to take something like that on — if there is enough interest (let me know!), I will host an online Google Hangout in a few weeks to help onboard people with contributing code.
2. Right now Emojitracker is paid for 100% out of pocket by me. While I have received some donations, the full amount of donations over the ~3 years of the project would cover about one month of hosting. If Emojitracker is upgraded to support newer emoji, those costs will go up. If you want to see it stick around, please donate and help cover the cost of hosting. Or if you work for Heroku, and are interested in sponsoring our hosting, I’d love to talk to you.
3. In my fantasy world, I’d raise enough money to pay an intern a fair wage to hack on Emojitracker as well. (Note I don’t need to make any money myself, but I am uncomfortable with exploiting the labor of others…) It’s too late to apply for Google Summer of Code now, but this is something I plan to look into for the future. Any similar arrangements people can recommend would be appreciated!