Mesosphere: A New Kind of Operating System
by Sunil Shah
After finishing the Master of Engineering program in May 2014, I joined Mesosphere, a San Francisco startup that is building the future of datacenter software. Mesosphere is one of the hottest infrastructure startups, having recently raised $36 million in series B fundraising and having announced our flagship product, the DataCenter Operating System. The core of our business is an open source project called Apache Mesos, a cluster management system that was developed at UC Berkeley’s AMPLab and now runs much of Twitter.com and many other companies’ datacenters.
Most MEng students should be familiar with the field of “big data.” Adoption of big data technologies is accelerating as businesses start to wake up to the powerful tools available. However, running a big data system like Hadoop (possibly the most popular) is expensive. A small Hadoop cluster can be a hundred nodes, and large clusters run into thousands of nodes. Often these clusters will work hardest at night, running machine learning or information retrieval algorithms while users are asleep; during other periods, they are often significantly under utilized and money is wasted.
Mesos and other cluster managers provide a uniform abstraction over a datacenter. Instead of statically allocating a thousand nodes to your Hadoop cluster and a thousand to other services, Mesos handles what runs where and how. This allows you to co-locate services in a way that changes over time. For example, a busy consumer website might have its heaviest load during the day. With Mesos, you can dynamically scale back the number of compute nodes devoted to the website at night and use that capacity for other services like, for example, Hadoop.
I joined Mesosphere as employee 21 in June 2014. We’ve grown to about three times the size since, with plans to grow to over a hundred by the end of the year. Since starting, I’ve worked on four different projects and played what might typically be four different roles in a more established company (engineer, recruiter, technical writer, and evangelist). Historically a Java programmer, I’ve since dabbled in Go, Scala, Python and Ruby!
As one of the more junior employees by experience, it’s been fantastic working with people who are all dedicated to the success of the company. The trait I appreciate most is that my colleagues are incredibly approachable and proactively help each other out. Nine months into the job, and I find that I’m still learning something new everyday.
With a community driven product like Mesos, there’s significant opportunity to get involved in the Bay Area meetup scene. Since joining I’ve given two presentations, one of which was attended by over a hundred engineers at Google’s HQ (I’ve never been so thankful for E295 Capstone Integration, AKA Communication for Engineers). Another great project was building out the backend for our product to let users easily deploy Mesos clusters on Google’s cloud product. As the only backend engineer on this project, I went from ground zero to shipping a working backend within six weeks. Hint: coffee. A lot of coffee.
While joining such a small company was something of a risk (especially as an international student), it’s been the most rewarding job I’ve held yet and I’m looking forward to the future aboard the Mesosphere rocketship!
