Talking Aurora at Mesoscon Europe
Video of my talk and a few thoughts on the conference.
I was privileged to speak at Mesoscon Europe last month about our DevOps work at Chartbeat with Apache Aurora. For more information about how we use Aurora and Mesos you can read the series of posts we’ve been writing.
First the video and then some comments and thoughts about the conference.
A few things stood out for me about the conference, aside from Prague being a great place to visit. It was exciting to see several presentations from companies building creative in-house solutions around Mesos. It’s always an exciting time in the evolution of a piece of software when the users are giving the presentations rather than the engineers who wrote it. This shows a level of adoption has been reached where users feel comfortable sharing their successes with the product. I saw great presentations from Yelp, Criteo, Netflix and Houghton Mifflin along with those from engineers at Mesosphere and other vendors.
We certainly fall into the user camp, and by the show of hands at the beginning of my talk it was clear that nobody knew who we were. To be fair, I think I lost a lot of people to the other talk at the same time “WTF My Container Just Spawned a Shell”… even I was bummed to miss that one. In our defense, we tend to be a must see talk at publishing industry conferences where our data scientists and product managers show off how our audience insight tools are used to help major news organizations drive reader engagement and the research we do with our data.
This was my first talk outside of company events or meet-ups and I was nervous… not really about public speaking, but about how the content of the talk would be received. This was really the first time we were sharing (outside of blog posts) the work we’ve been doing with Mesos and I was worried that nobody would find it interesting, or worse, think we had done things “wrong”.
When I put up the slide showing what we do and how much traffic we get, I saw more than a few people look up from their laptops and start to pay attention. I think I rushed a bit, there was a lot of content to cram into 45 minutes. Several folks in the audience came up to me afterwards and asked more about the project or just told me that they enjoyed it. They asked about specific choices — were we using nerve at all or just synapse? One engineer who also spoke at the conference told me he would have strongly considered Aurora over Marathon if it had worked the way we described out of the box.
One of the committers to the Mesos project found me outside the room afterwards and we had a long talk about why so many companies are ignoring Aurora. It definitely takes a lot more work to get Aurora running compared to Marathon, especially with DC/OS making installing Marathon really a point and click exercise and an easy choice. For many users, that’s actually the right choice to make, but for us, and many others like us, it’s not. Aurora is a lot more flexible than Marathon and I think I showed many people that I might be right, or at least not wrong.
Hopefully next time my talk won’t be up against WTF-MCJSAS.
I go to conferences for inspiration more than anything else, I don’t stress about going to every single talk but I do try to find opportunities to have good conversations with other engineers and to find a few big ideas. Conferences are NOT good places to go to learn the details about a new technology, they’re for deciding what you want to spend your time doing when you get home. My takeaway big ideas from this conference were writing custom frameworks, and starting to contribute more of our work back to the OSS community.
Quite a few companies have been writing their own frameworks with the v1 api, but the new HTTP Scheduler API makes writing custom frameworks much easier now. I spent my first Hack Week back working on some prototypes with pymesos as well as starting to write a framework client in clojure.
In terms of writing more OSS, we do have quite a few projects on our public repo, but I want us to start contributing to larger projects like Aurora. It’s great that companies like Mesosphere and Confluent can maintain the core of projects that data engineering relies on but the real innovation still tends to happen inside the companies that are the end users of these products. I think we have a lot of great tech to contribute and we will. I think it’s very important for the future of Mesos that there are several great framework choices available.
Finally, a plug for conference activities in the evening. I made an excellent choice by going to the drink-time “Town Hall” chat. These were small groups broken up around a few themes. I went for the one that sounded the most generic (I think it was called core mesos internals or something like that). It turned out that I chose the one with several of the main committers to the Mesos project and I got to hear a lot about what’s working, what’s not and what they’re working on. I also got to share what we find compelling about Aurora. It was a great conversation between a dozen engineers. I strongly encourage anyone who attends smaller conferences to attend these evening events. They’re becoming much more common and can be a great opportunity to meet interesting people and to hear their ideas.
