Facebook VP on scaling open source and building infrastructure for drones
Facebook Vice President of Engineering Jay Parikh has overseen some of the company’s most ambitious technology projects over the past few years, from launching the Open Compute Project to delivering internet access via unmanned aerial vehicles.
In this interview, he discusses — among other things — how the upcoming @Scale conference (which Facebook helps put on) is helping bring leading engineers together to share their code and ideas, and how Facebook’s global scale means it keeps finding new challenges to tackle.
SCALE: What’s the history of the @Scale conference, and the rationale behind it?
JAY PARIKH: We’ve actually been doing these @Scale conferences for several years, probably 2 or 3 years now, but they were on a very small scale. We found space here on campus; threw some stuff together; found some other companies that were excited to share what they were doing; invited a bunch of people; and just did these very informally, almost as meetups, but around a specific topic, such as mobile, or security, or networking.
We really tried to focus on presenters and companies doing interesting things and that were going to be open about it — that were either open-sourcing the code or were willing to talk about the challenges they faced, the solutions they found, how they learned and built what they’re building, and the challenges ahead. Also, to frame what the future looks like for that system or that product that they’re working on.
Last year was the first year we said, “You know what? Maybe we should just put this together and have an annual event that allows us to bring multiple tracks together. Mobile and web and data and security. Let’s see if we can bring a bunch of these tracks all together and bring more engineers.”
Because what you end up doing as an engineer, you have to worry about a lot of these things all at the same time. You’re doing mobile development, but then you need to understand how all of the metrics flow through the system and how you make decisions about building a better product. Or you’re working on a web product and you need to make it mobile and you have to deal with iOS and Android, and this and that.
A lot of what we’ve found practically was that engineers cared (not surprisingly, I guess) about many different topics all at the same time, and that mashing it up might be useful. We tried it and we got a ton of positive feedback.
“I think we’re all facing scale challenges. The number of zeroes may be different between the companies, but they’re all interesting, they’re all fascinating problems.”
How do you compare @Scale to the other tech meetups and conferences that are help regularly around Silicon Valley and elsewhere?
I think part of it is that it’s very focused on scale and the scalability challenges that customers have. There are lots of meetups and conferences that are around a specific technology, so you can go to the Hadoop Summit or the HBaseCon, or you can go to this conference and learn about Mesos or this framework or whatever. There’s a lot of great domain-specific things, but I have not yet seen conferences that allow engineers to see the mashup of lots of these things and how they can use a combination of things to solve the problems that they’re facing.
But it’s also not vendors. We don’t have vendors here pitching people. This is about companies like Airbnb, and us, and Google, and Twitter, and LinkedIn, and Uber, and Microsoft, and WhatsApp, and Instagram. We want all of these companies to be talking and sharing. We’re really trying to orient this around engineers who are building stuff talking to other engineers building stuff.
How difficult is it to get Google, Facebook, Twitter and everyone talking broadly about their core technologies, given the competition at higher levels of the business?
I think it helps once we align around what we’re trying to do here, that this is not a Facebook-specific event. This is an independent thing and we really want people to see that while we provide a bunch of coordination help, it’s not a Facebook thing, it’s not a Google thing, and it’s not a Twitter thing. It’s about building this community of companies and people who are passionate about these challenges, learning from each other and networking with each other, and hopefully being able to help each other.
I generally think that a lot of these problems that people will be presenting on, they’re not unique. There may be some uniqueness to the problem, but every company, or lots of companies, face distributed key-value stores or some security thing, or developing mobile apps to be fast. All of us are facing these challenges.
I do think people want to know what’s in it for them and what they get out of it. But I think, honestly, most engineers are extremely excited about coming in and showing off the cool stuff they’re building, and I think this is a fantastic venue where they get to come in and geek out and talk about what they’re building, why they’re building it and how it all works.
“Mark [Zuckerberg] built Facebook on PHP, MySQL and then Memcached, and these were all open-source projects when he started Facebook in his dorm in 2004.”
Expanding the open source footprint
From the engineering perspective, aside from the conference, it seems like there are a broader swath of companies — beyond Facebook, Google and Twitter, for example — that have grown up fast that are now contributing things to the broader community.
I think that one of the things that we look for, and who we think about inviting or wanting to present here, is you have to be philosophically aligned to “open is better” and sharing and contributing and being part of this community. The folks from Pixar are going to be there. They’ve announced that they’re going to open-sourcing stuff, too.
I think a lot of these companies, including Facebook, grew up fast because of open-source and because of sharing and being open. Mark [Zuckerberg] built Facebook on PHP, MySQL and then Memcached, and these were all open-source projects when he started Facebook in his dorm in 2004. We still obviously are big proponents of open source and do a lot of it.
Uber’s the same way. Twitter’s the same way. LinkedIn’s the same way. Pinterest is the same way. We all share that same set of core values, with “open” being on the top of it.
“There’s lots of things that we can do to bring sharing video to the same level of sharing as text or photos.”
I know Facebook has inspired a lot of companies and a lot of companies use Facebook tech. Does Facebook adopt any open source technologies coming out of other places?
I think both are true. For example, there’s a lot of contributions that we get into projects that we’ve open-sourced. We get a ton of really, really awesome features, enhancements, bug fixes, into things like HHVM, React and othr projects that we’ve worked on. But we absolutely keep our eye out and dabble and use lots of stuff.
Over time, it just depends where in the business you are. We don’t have any prescription here which is like, “Well, we can never use somebody else’s open-source project.” There is no rule or anything like that; we want the best solution at all times.
If that means that we can pick up something that somebody else did and use it here, likely we have to make it work in our environment and make it work at our scale, but that’s part of the assessment and what we evaluate when we’re looking at a project that may not have been built here.
Onto bigger, better things …
It seems like Facebook has solved, to some degree, a lot of the main core components of running a large web service. What’s left to tackle after compute, storage, networking and all the other projects you’ve done?
I think it’s all of it, still. There’s a lot of the core that still can, and should, get a lot better. It can get faster, it can get more efficient, it can provide more capabilities.
I think one of the challenges that we see,very specific to Facebook, is that with the broad portfolio of applications now here, it’s not just about the main Facebook app — it’s about Instagram, it’s about Whatsapp, it’s about Messenger. It’s about other things, like that Moments app or Mentions.
Really, just the needs of these other use cases and applications within Facebook is pushing new, different and difficult demands or requirements onto the core infrastructure. Some of that is iterating on what we already have, some of that is still a work in progress and will likely result in new stuff being invented.
“My hope is that we are able to deliver a very rich experience to [everyone], including videos, photos and — some day — virtual reality and all of that stuff.”
For example, we’re doing a bunch of work on deep learning, with the artificial intelligence team. There’s a lot of innovative, hard software problems that are being solved or being investigated and, believe it or not, there’s software stuff there that we will open-source. And we want to optimally run that on a physical infrastructure. What does that mean in terms of the network, in terms of the datacenter, in terms of the servers that this software runs on? Well, there may be a better way to do it than what we’ve done for the standard Facebook website, news feed and serving photos.
The same is true for things like video. I think we’re at a very early phase of growing video to be a bigger component of the product experience across our family of apps. What that may do from an infrastructure perspective is beyond just getting a bunch of bandwidth because you’re going to serve more bits to people. There’s lots of things that we can do to bring sharing video to the same level of sharing as text or photos.
I also think there is the further-out there stuff that we’re working on. It’s the stuff outside of the four walls of the datacenter, focused on connecting the people who don’t have internet connectivity today. That’s where our unmanned aircraft program is, working on communications technologies involving free space optics or lasers and other RF technologies. There’s a bunch of work we’re doing there, and there are some really neat ways to leverage the science and the physics that we’ve figured out inside the datacenter to potentially help us get people connected to the internet, where we can reduce the cost or increase the capacity by an order of magnitude or more.
“We’re operating at a scale that I think very few people do so. We obviously have to make these things work in our environment, then we try to share as much of this back as we can.”
How much of the open internet work falls under your purview? Is that something that your team works on fairly regularly?
It’s all in the infrastructure team. There are two components to Internet.org. There’s the business and the marketing side, which is around free basic services and offering the app that gets people to use it and to experience the internet. Then there’s the technological challenges of, “OK, there’s a village of 10,000 people who live 100 kilometers away from any sort of signal, or only have 2G, or don’t have access to fiber.” The unmanned aircraft, laser technology — that’s all part of Connectivity Lab, which is within the infrastructure team.
You talked about making video more part of the internet experience. Can you get to a point where people benefitting from free internet can be part of that, or do they have to settle for Facebook v0.5 or some scaled-down version of the web?
I think it’s still early and we’re figuring it out. The goal right now is to get everybody access to the internet as fast as possible. Keep in mind, 2.5 billion to 3 billion people are already online and part of the internet, or use the internet. There’s also a couple of billion people who aren’t online, but have technical access to the technology, but they just don’t use the internet. There’s some social, economic, or other reason why they have not on-boarded or on-ramped to the internet, and that’s where our free basic services and the other half of Internet.org is really focused on.
Then there’s the last 10 to 15 percent of the world’s population that doesn’t have access to the internet, and there’s a whole set of challenges there. The Connectivity Lab team is very focused on the technical challenges of reaching those people who are typically in the more rural, unconnected parts of the world. I think that we need to get them access.
They likely will not have multiple tens of megabits, or hundreds of megabits of access direct to their phone on Day One, so they’re not going to get to the level of connectivity that we have, for example, here in the Bay Area. But quickly, we want to get them access and we want to keep making it better and better and better.
My hope is that we are able to deliver a very rich experience to them, including videos, photos and — some day — virtual reality and all of that stuff. But it’s a multi-, multi-, multi-year challenge, and I don’t see any end in sight right now.
From an engineering perspective, what’s the challenge your team faces in bringing together all the companies — and associated technology stacks and engineering principles — you acquire?
I’d say it’s all kind of hard. There’s a few things that I’d say, maybe from a principle perspective, that are important and I think maybe are different in terms of how we think about these acquisitions. It’s all about continuing to focus on the priorities of that product.
For example, when we acquired Instagram a couple years back, the focus was very well known in the company that we need to do whatever it takes to help Instagram achieve their product goals, growth goals and all that kind of stuff. We really didn’t come in here with, “Hey, we’re big, bad Facebook, you’re going to do it our way.”
It was more like, “Hey, how can we help? What are things that you guys are working on that are challenging for you? Let us show you our toy store of all of these things that we’ve built. Can you use any of these things? Then we can help you.”
So we kicked off this project that I code named “Instagration.” There was no sort of forced plan, there was no, “Oh, we have to get off of AWS because it’s costing us too much money” or “You have to use our stuff.” It was like, “What’s right for you and when is it right for you?”
They were immediately able to identify some problems. They had some scale issues around some of the site-integrity stuff, detecting spam and other kinds of not-great content coming into the system. They were able to plug into our very sophisticated and very highly scaled systems to offload that problem to a system and a team here that is world-class.
We just kept chipping away at it and within a couple years, we made a real plan and got them off of AWS, and now they roll on our own infrastructure. But it was really about making sure we were there to support the overall goals of the product versus, “You have to integrate because … you just have to integrate.”
“You could probably assume that I’m never satisfied with anything.”
Is there a point where Facebook outgrows some of the open source projects that it has created or developed on or contributed a lot to over the years?
Absolutely, all the time. We have dozens of examples of this. Mark built this site on Apache and then we outgrew that. We did the HipHop Compiler, we outgrew that and did HHVM. We looked at HTML 5, did that, and that didn’t work out for us so we went native with apps. Now we’re looking at React.
Some of this stuff is us building on things, like our version of Hadoop is very different than what you download off of the web today. Memcached, we outgrew that, so we did a bunch of enhancements to it. We built things like McRouter, and we’ve built TAO.
We’re operating at a scale that I think very few people do so. We obviously have to make these things work in our environment, then we try to share as much of this back as we can.
I don’t think it’s just true of us. I think this happens at Twitter. It happens at LinkedIn. It happens at Pinterest. I think we’re all facing scale challenges. The number of zeroes may be different between the companies, but they’re all interesting, they’re all fascinating problems.
Assessing the Open Compute Project
I think the Open Compute Project is one of the biggest projects Facebook has worked on. Are you satisfied with that, in the sense that the industry at large has received the message that Open Compute was created to send?
You could probably assume that I’m never satisfied with anything. I think we’ve done something very difficult, I think we’ve kicked off something that is very hard to do and I’m excited about what we’ve accomplished. But I’m actually more excited about where this goes, especially with the progress we’ve made this year and late last year with the top-of-rack switch and FBOSS, and with 6-pack. So now we have the networking piece, we have storage and we have compute. The datacenters are a piece of it too. I do think that there’s a lot more to do, though.
It’s a big industry, worth billions of dollars. Now we have HP there, we have Microsoft there. We have other companies, like Apple, who are using and evaluating this gear. It’s all about driving more and more adoption of the best practices, designs, innovations and technology that’s coming out of this community. That’s the work ahead of us.
We have some really good proof points and examples where the Open Compute Project’s work seems to really work and matter in the world. But I still think it’s early. We’re only a couple of years into this thing and like I said, I’ll never be happy with this. But I’m excited.
And we’re actually using a lot of the same principals as we think about Connectivity Lab. As we build these unmanned aircraft and the communications systems involving lasers, or RF, we don’t want to operate all this stuff. We’re not trying to be our own carrier, or our own operator. We really want to inspire the industry — other companies and vendors — to take this technology and get it out there, to deploy it, run it and operate it. We just want the industry to move faster on connecting people.
“Industry is, in a lot of places, settled into a lot of incremental improvements, and it’s not good enough anymore.”
That last point is what I was getting at. Do you think the industry, the vendors, have heeded the writing on the wall?
I don’t think it’s the writing on the walls, I just think it’s the times are different now. And we all need to do better in terms of making the technology we bring to market cheaper and better. We have to rethink the business models. I think there’s a lot of opportunity for partnerships here. We’ve certainly seen a crazy amount of interest on the Connectivity Lab stuff, and people jazzed about the future here.
And I think, frankly, that Facebook is in a very unique position to do this given the number of people who rely on us for our portfolio of applications; our relationships in the industry; the proof points that projects like OCP are open; and our commitment to work on these things for a long time. They’re not going to change in a quarter. They’re going to take awhile to build a community, to really inflect the thinking and to get the technological advancements to happen at the magnitude that you want.
Industry is, in a lot of places, settled into a lot of incremental improvements, and it’s not good enough anymore. That’s what we’re really trying to push and inspire. We’ve assembled, and we are continuing to assemble, what I hope is a world-class team of experts across all of these different disciplines — whether it be lasers and opticals, kernel engineering, computing or chip design. We have to have a team here who can pull these different disciplines together, come up with these innovations, and work with the community to try to change roadmaps and product lines.
