Preparing for the Oscars 2016 on Twitter — #realtime, #oscarsdata

It's 2016, and whether big data is a buzzword or a movement, crunching numbers is part of our daily life. The red carpet is about to start, will we be lazy on a couch with popcorn!? Not this time…

This is what we'll be doing, and you can join us by following #oscarsdata!

We're going to watch streams of real time data, crunch numbers, and monitor what's trending on Twitter.

We've got a team of 10 people from several of Silicon Valley's hottest startups representing a dozen code bases. Our day will be filled with good conversations around movies, food and wine. And of course, numbers.

Our "production" system is Google Cloud, with Dataflow at its core, PubSub, Datastore and AppEngine to serve real time data. The front end is done in React with CanvasJs for real time charting, hosted on GitHub, where we'll store the "final results", like we did last year.

Behind the scenes, we also have a cluster of 7 beefy machines, trying to reproduce the same results on premise. We're experimenting with tons of tools, including Kafka, Spark, Flink and MemSQL. And, time and wine permitting, it will also be running on Docker on this same Mac I'm using to write this post.

Sound exciting? Follow our day on #oscarsdata, and give a shout out if you're inclined.